Image processing apparatus and image processing method

ABSTRACT

The present technology relates to an information processing apparatus and an image processing method that make it possible to accurately reproduce a blur degree of an optical lens. A ray generation section generates rays to be incident to a virtual lens having a synthetic aperture configured from a plurality of image pickup sections that pick up images of a plurality of visual points from a real space point in a real space. A luminance allocation section allocates a luminance to rays remaining as a result of a collision decision for deciding whether or not the rays collide with an object before the rays are incident to the virtual lens. The present technology can be applied to a light field technology for reconstructing, for example, images picked up using various optical lenses from images of a plurality of visual points.

TECHNICAL FIELD

The present technology relates to an information processing apparatusand an image processing method, and particularly to an image processingapparatus and an image processing method that make it possible, forexample, to accurately reproduce a blur degree of an optical lens.

BACKGROUND ART

A light field technology is proposed which reconstructs, for example, animage for which refocusing is performed, namely, an image that looks asif image pickup were performed changing the focus position of an opticalsystem or a like image from images of a plurality of visual points (forexample, refer to PTL 1).

CITATION LIST Patent Literature

-   [PTL 1]-   JP 2013-238927A

SUMMARY Technical Problem

For the light field technology, it is demanded to accurately reproduce ablur degree appearing on an image when image pickup is performed usingan actual optical lens.

The present technology has been made in view of such a situation as justdescribed and makes it possible to accurately reproduce a blur degree ofan optical lens.

Solution to Problem

The information processing apparatus of the present technology is animage processing apparatus including a ray generation section configuredto generate rays to be incident to a virtual lens having a syntheticaperture configured from a plurality of image pickup sections that pickup images of a plurality of visual points from a real space point in areal space, and a luminance allocation section configured to allocate aluminance to rays remaining as a result of a collision decision fordeciding whether or not the rays generated by the ray generation sectioncollide with an object before the rays are incident to the virtual lens.

The information processing method of the present technology is an imageprocessing method including generating rays to be incident to a virtuallens having a synthetic aperture configured from a plurality of imagepickup sections that pick up images of a plurality of visual points froma real space point in a real space, and allocating a luminance to raysremaining as a result of a collision decision for deciding whether ornot the rays collide with an object before the rays are incident to thevirtual lens.

In the image processing apparatus and the image processing method of thepresent technology, rays to be incident to the virtual lens having thesynthetic aperture configured from the plurality of image pickupsections that pick up images of the plurality of visual points aregenerated from the real space point in the real space, and a luminanceis allocated to rays remaining as a result of the collision decision fordeciding whether or not the rays collide with an object before the raysare incident to the virtual lens.

It is to be noted that the image processing apparatus may be anindependent apparatus or may be an internal block configuring a singleapparatus.

Further, the image processing apparatus can be implemented by causing acomputer to execute a program, and the program can be provided bytransmitting the same through a transmission medium or by recording thesame on a recording medium.

Advantageous Effect of Invention

According to the present technology, for example, a blur degree of anoptical lens can be reproduced accurately.

It is to be noted that the effect described here is not necessarilyrestrictive and may be any of effects described in the presentdisclosure.

BRIEF DESCRIPTION OF DRAWINGS

FIG. 1 is a block diagram depicting an example of a configuration of anembodiment of an image processing system to which the present technologyis applied.

FIG. 2 is a plan view depicting an example of a configuration of animage pickup apparatus 11.

FIG. 3 is a block diagram depicting an example of a configuration of animage processing apparatus 12.

FIG. 4 is a flow chart illustrating an example of a process of the imageprocessing system.

FIG. 5 is a block diagram depicting an example of a configuration of aparallax information generation section 31.

FIG. 6 is a view illustrating an example of generation of a referenceparallax map by a reference parallax map generation section 41.

FIG. 7 is a view illustrating an example of generation of a parallax mapof a peripheral image PL#i.

FIG. 8 is a view illustrating interpolation of a parallax to anon-registration area of the parallax map of the peripheral image PL#i.

FIG. 9 is a view illustrating an example of generation of a multilayerparallax map.

FIG. 10 is a flow chart illustrating an example of a process ofgeneration of the reference parallax map and the multilayer parallax mapby the parallax information generation section 31.

FIG. 11 is a view depicting an example of an actual image obtained bypicking up an image of a predetermined image pickup object using anactual optical lens.

FIG. 12 is a view depicting an example of an emulation image obtained bythe image processing apparatus 12.

FIG. 13 is a view illustrating a principle by which a clear blur is notreproduced when a saturated pixel restoration process is not performed.

FIG. 14 is a view illustrating a principle by which a clear blur isreproduced by performing the saturated pixel restoration process.

FIG. 15 is a view illustrating a first acquisition method for acquiringa standard luminance picked up image PL#i and a low luminance picked upimage PH#i.

FIG. 16 is a block diagram depicting a first example of a configurationof a saturated pixel restoration section 33.

FIG. 17 is a flow chart illustrating an example of the saturated pixelrestoration process performed by the saturated pixel restoration section33.

FIG. 18 is a block diagram depicting a second example of a configurationof the saturated pixel restoration section 33.

FIG. 19 is a flow chart illustrating an example of the saturated pixelrestoration process performed by the saturated pixel restoration section33.

FIG. 20 is a plan view depicting another example of a configuration ofthe image pickup apparatus 11.

FIG. 21 is a block diagram depicting a third example of a configurationof the saturated pixel restoration section 33.

FIG. 22 is a view illustrating an example of correction of a parallax ofa parallax map.

FIG. 23 is a flow chart illustrating an example of the saturated pixelrestoration process performed by the saturated pixel restoration section33.

FIG. 24 is a flow chart illustrating an example of a process foracquiring a pixel value of a noticed pixel of an HDR (High DynamicRange) picked up image at a noticed visual point.

FIG. 25 is a view illustrating an outline of a lens emulation process ofa lens emulation section 35.

FIG. 26 is a view illustrating a light condensing process by an actualoptical lens and a light condensing process of the lens emulationprocess.

FIG. 27 is a block diagram depicting an example of a configuration of anincident ray reproduction section 36.

FIG. 28 is a view illustrating a real space point.

FIG. 29 is a view illustrating a determination method for determining areal space point using a multilayer parallax map.

FIG. 30 is a view illustrating an example of generation of raysperformed by a ray generation section 102.

FIG. 31 is a view illustrating collision decision performed by acollision decision section 103 and allocation of luminance to raysperformed by a luminance allocation section 104.

FIG. 32 is a view schematically depicting a maximum number of dataobtained by an incident ray reproduction process performed by theincident ray reproduction section 36.

FIG. 33 is a flow chart illustrating an example of the incident rayreproduction process performed by the incident ray reproduction section36.

FIG. 34 is a flow chart illustrating an example of a process forcollision decision.

FIG. 35 is a flow chart illustrating an example of a process for rayluminance allocation.

FIG. 36 is a view illustrating lens information generated by anemulation lens information generation section 37.

FIG. 37 is a view illustrating a real space point that is a target forgeneration of lens information and a focus position.

FIG. 38 is a view depicting an example of a PSF (Point Spread Function)intensity distribution of an optical lens.

FIG. 39 is a view illustrating an example of a method for generating aPSF intensity distribution.

FIG. 40 is a view schematically depicting a PSF intensity distributiongenerated by the emulation lens information generation section 37.

FIG. 41 is a view schematically depicting an image plane pitch generatedby the emulation lens information generation section 37.

FIG. 42 is a view illustrating an example of a method for generating PSFangle component information.

FIG. 43 is a view illustrating an example of a method for generating PSFangle component information.

FIG. 44 is a view illustrating details of the PSF angle componentinformation.

FIG. 45 is a view schematically depicting PSF angle componentinformation generated by the emulation lens information generationsection 37.

FIG. 46 is a view illustrating image plane shift information.

FIG. 47 is a view illustrating an example of a method for generating PSFangle component information.

FIG. 48 is a view schematically depicting image plane shift informationgenerated by the emulation lens information generation section 37.

FIG. 49 is a block diagram depicting an example of a configuration ofthe emulation lens information generation section 37 that generates lensinformation.

FIG. 50 is a flow chart illustrating an example of an emulation lensinformation generation process performed by the emulation lensinformation generation section 37.

FIG. 51 is a view illustrating an outline of a light condensing processperformed by a light condensing processing section 38.

FIG. 52 is a view illustrating an example of a process for determiningan image formation value from within the light condensing process.

FIG. 53 is a view illustrating a different example of a process fordetermining an image formation value from within the light condensingprocess.

FIG. 54 is a view illustrating an example of a process for adding (adistribution of) an image formation value to a virtual sensor fromwithin the light condensing process.

FIG. 55 is a block diagram depicting an example of a configuration ofthe light condensing processing section 38.

FIG. 56 is a flow chart illustrating an example of the light condensingprocess performed by the light condensing processing section 38.

FIG. 57 is a flow chart illustrating an example of a ray additionprocess.

FIG. 58 is a view illustrating an outline of reduction of theinformation amount of lens information.

FIG. 59 is a view depicting a particular example of a lens informationgeneration axis.

FIG. 60 is a block diagram depicting an example of a configuration ofthe emulation lens information generation section 37 where lensinformation is generated only for an information point of the lensinformation generation axis.

FIG. 61 is a flow chart illustrating an example of the emulation lensinformation generation process performed by the emulation lensinformation generation section 37.

FIG. 62 is a view illustrating an example of a light condensing processperformed using lens information generated for (a real space pointcorresponding to) an information point of the lens informationgeneration axis.

FIG. 63 is a view depicting an example of a light condensing process bya ray rotation method.

FIG. 64 is a block diagram depicting an example of a configuration ofthe light condensing processing section 38 that performs a lightcondensing process using lens information generated for the lensinformation generation axis.

FIG. 65 is a flow chart illustrating an example of the light condensingprocess performed by the light condensing processing section 38.

FIG. 66 is a flow chart illustrating an example of the ray additionprocess.

FIG. 67 is a view illustrating a method for determining an image planeshift position that is an image formation position on the virtual sensoron which rays before rotation form an image where the light condensingprocess by the ray rotation method is performed.

FIG. 68 is a view illustrating interpolation of image plane shiftinformation in a juxtaposition direction of information points.

FIG. 69 is a view illustrating interpolation of image plane shiftinformation in a parallax direction.

FIG. 70 is a view depicting an example of an emulation image obtained asa result of the lens emulation process by the lens emulation section 35.

FIG. 71 is a view depicting a different example of an emulation imageobtained as a result of the lens emulation process by the lens emulationsection 35.

FIG. 72 is a block diagram depicting an example of a configuration of anembodiment of a computer to which the present technology is applied.

DESCRIPTION OF EMBODIMENT

<Embodiment of Image Processing System to which Present Technology isApplied>

FIG. 1 is a block diagram depicting an example of a configuration of anembodiment of an image processing system to which the present technologyis applied.

In FIG. 1, the image processing system includes an image pickupapparatus 11, an image processing apparatus 12 and a display apparatus13.

The image pickup apparatus 11 picks up an image of an image pickupobject from a plurality of visual points and supplies picked up imagesof the plurality of visual points obtained as a result of image pickupto the image processing apparatus 12.

The image processing apparatus 12 performs an image process using thepicked up images of the plurality of visual points from the image pickupapparatus 11 to generate an emulation image similar to that where animage of the image pickup object is picked up using an emulation lensthat is an optical lens of an emulation target and supplies theemulation image to the display apparatus 13.

The display apparatus 13 displays the emulation image from the imageprocessing apparatus 12.

The emulation image is an image by which, for example, a blur degreegenerated in an image of an image pickup object picked up using anoptical lens removably mounted on a single-lens reflex camera or amirror-less camera is reproduced faithfully. Therefore, the user canenjoy the blur degree of such an expensive optical lens withoutpurchasing the expensive optical lens.

It is to be noted that, in FIG. 1, the image pickup apparatus 11, theimage processing apparatus 12 and the display apparatus 13 configuringthe image processing system can be built in an independent apparatussuch as, for example, a digital (still/video) camera or a portableterminal such as a smartphone.

Further, the image pickup apparatus 11, the image processing apparatus12 and the display apparatus 13 can be individually built in anindependent apparatus.

Further, arbitrary two and a remaining one of the image pickup apparatus11, the image processing apparatus 12 and the display apparatus 13 canbe individually built in an independent apparatus.

For example, the image pickup apparatus 11 and the display apparatus 13can be built in a portable terminal the user possesses and the imageprocessing apparatus 12 can be built in a server on a cloud.

Further, blocks of the image processing apparatus 12 can be built in aserver on a cloud and the remaining blocks of the image processingapparatus 12 and the image pickup apparatus 11 and the display apparatus13 can be built in a portable terminal.

<Example of Configuration of Image Pickup Apparatus 11>

FIG. 2 is a plan view depicting an example of a configuration of theimage pickup apparatus 11.

The image pickup apparatus 11 includes a plurality of camera units 21_(i) and picks up picked up images of a plurality of visual points bythe plurality of camera units 21 _(i).

In FIG. 2, the image pickup apparatus 11 includes a plurality of, forexample, seven camera units 21 ₁, 21 ₂, 21 ₃, 21 ₄, 21 ₅, 21 ₆ and 21 ₇,and the seven camera units 21 ₁ to 21 ₇ are disposed on atwo-dimensional plane.

Further, in FIG. 2, for example, centering around the camera unit 21 ₁that is one of the seven camera units 21 ₁ to 21 ₇, the other six cameraunits 21 ₂ to 21 ₇ are disposed around the camera unit 21 ₁ so as toconfigure a regular hexagon.

Accordingly, in FIG. 2, the distance between (the optical axes of) anarbitrary one camera unit 21 _(i) (i=1, 2, . . . , 7) from among theseven camera units 21 ₁ to 21 ₇ and another camera unit 21 _(j) (j=1, 2,. . . , 7) nearest to the camera unit 21 _(i) is the equal distance L.

As the distance L between the camera units 21 _(i) and 21 _(j), forexample, approximately 20 mm can be adopted. In this case, the imagepickup apparatus 11 can be configured in the size of a card such as anIC (Integrated Circuit) card.

It is to be noted that the number of the camera units 21 _(i)configuring the image pickup apparatus 11 is not limited to seven and anumber equal to or greater than two but equal to or smaller than six ora number equal to or greater than eight can be adopted.

Further, in the image pickup apparatus 11, the plurality of camera units21 _(i) can be disposed so as to configure a regular polygon such as aregular hexagon as described above or can be disposed at arbitrarypositions.

Here, the camera unit 21 ₁ disposed at the center from among the cameraunits 21 ₁ to 21 ₇ is referred to also as reference camera unit 21 ₁ andthe camera units 21 ₂ to 21 ₇ disposed around the reference camera unit21 ₁ are referred to sometimes as peripheral camera units 21 ₂ to 21 ₇.

<Example of Configuration of Image Processing Apparatus 12>

FIG. 3 is a block diagram depicting an example of a configuration of theimage processing apparatus 12 of FIG. 1.

In FIG. 3, the image processing apparatus 12 includes a parallaxinformation generation section 31, a calibration data acquisitionsection 32, a saturated pixel restoration section 33, a lens design dataacquisition section 34 and a lens emulation section 35.

To the image processing apparatus 12, picked up images of seven visualpoints picked up by the camera units 21 ₁ to 21 ₇ are supplied from theimage pickup apparatus 11.

Here, as the picked up images of the seven visual points supplied to theimage processing apparatus 12 by the image pickup apparatus 11, standardluminance picked up images PL1 to PL7 and low luminance picked up imagesPH1 to PH7 are available.

A standard luminance picked up image PL#i is an image picked up in apredetermined exposure time period (at a shutter speed) (hereinafterreferred to also as standard exposure time period) estimated suitable,for example, upon image pickup by the camera unit 21 _(i). As thestandard exposure time period, for example, an exposure time period setby an automatic exposure function or the like can be adopted.

A low luminance picked up image PH#i is an image picked up in anexposure time period shorter than the standard exposure time period (ata shutter speed higher than the shutter speed corresponding to thestandard exposure time period) by the camera unit 21 _(i).

Accordingly, in the low luminance picked up image PH#i, roughly an imagepickup object that is reflected in the standard luminance picked upimage PL#i is reflected dark.

In the image processing apparatus 12, the standard luminance picked upimage PL#i is supplied to the parallax information generation section 31and the saturated pixel restoration section 33, and the low luminancepicked up image PH#i is supplied to the saturated pixel restorationsection 33.

The parallax information generation section 31 determines parallaxinformation using the standard luminance picked up image PL#i suppliedfrom the image pickup apparatus 11 and supplies the parallax informationto an incident ray reproduction section 36, an emulation lensinformation generation section 37 and a light condensing processingsection 38 hereinafter described that configure the lens emulationsection 35.

In particular, the parallax information generation section 31 performs aprocess for determining parallax information of each of the standardluminance picked up images PL#i supplied from the image pickup apparatus11 from the different standard luminance picked up image PL#j as animage process of the standard luminance picked up images PL#i of theplurality of visual points. Then, the parallax information generationsection 31 generates a parallax map in which the parallax information isregistered and supplies the generated parallax map to the lens emulationsection 35.

Further, the parallax information generation section 31 generates amultilayer parallax map hereinafter described in regard to the standardluminance picked up image PL1 picked up by the reference camera unit 21₁ from among the standard luminance picked up images PL#i and suppliesthe generated multilayer parallax map to the lens emulation section 35.

Here, as the parallax information, not only the parallax (disparity)itself but also the distance (depth) corresponding to the parallax canbe adopted. In the present embodiment, as the parallax information, theparallax, for example, from between the parallax and the distance isadopted.

The calibration data acquisition section 32 acquires a distortion valueand a shading coefficient of the optical lens of each of the cameraunits 21 ₁ to 21 ₇ as calibration data.

Here, the calibration data is stored, for example, in a memory notdepicted or is provided from a server or the like on the Internet. Thecalibration data acquisition section 32 acquires calibration data fromthe memory or the server on the Internet and supplies the acquiredcalibration data to the parallax information generation section 31 andthe incident ray reproduction section 36.

The parallax information generation section 31 and the incident rayreproduction section 36 perform a calibration process for making pickedup images picked up by the peripheral camera units 21 ₂ to 21 ₇(standard luminance picked up images PL2 to PL7 or HDR picked up imagesHD2 to HD7 hereinafter described) coincide with a picked up image pickedup by the reference camera unit 21 ₁ (a standard luminance picked upimage PL1 or an HDR picked up image HD1 hereinafter described) using thecalibration data supplied from the calibration data acquisition section32.

In particular, the parallax information generation section 31 and theincident ray reproduction section 36 perform, using the calibrationdata, a calibration process for correcting picked up images picked up bythe peripheral camera units 21 ₂ to 21 ₇ to a picked up image that maybe obtained if image pickup is performed using the reference camera unit21 ₁ in place of the peripheral camera units 21 ₂ to 21 ₇.

Then, the parallax information generation section 31 and the incidentray reproduction section 36 perform a process for the picked up imagepicked up by the peripheral camera units 21 ₂ to 21 ₇ using the pickedup images after the calibration process.

It is to be noted that the calibration process is not hereinafterdescribed in order to simplify the description.

The saturated pixel restoration section 33 restores a pixel value of asaturated pixel whose pixel is saturated from among pixels of thestandard luminance picked up image PL#i supplied from the camera unit 21_(i) using the low luminance picked up image PH#i supplied from thecamera unit 21 _(i).

The saturated pixel restoration section 33 converts the standardluminance picked up image PL#i into a picked up image HD#i of a higherdynamic range than that of the standard luminance picked up image PL#i(in which the number of bits allocated to a pixel value is greater) bythe restoration of the pixel value of the saturated pixel and suppliesthe picked up image HD#i to the incident ray reproduction section 36.

It is to be noted that, in the saturated pixel restoration section 33,the picked up image HD#i having a higher dynamic range than that of thestandard luminance picked up image PL#i can be supplied not only to theincident ray reproduction section 36 but also to the parallaxinformation generation section 31.

In this case, in the parallax information generation section 31, animage process for determining parallax information can be performedusing the picked up image HD#i having the high dynamic range in place ofthe standard luminance picked up image PL#i. Where the parallaxinformation is determined using the picked up image HD#i having the highdynamic range, the parallax information can be determined with a higherdegree of accuracy.

Here, the picked up image HD#i of a high dynamic range obtained by therestoration of the pixel value of a saturated pixel is referred to alsoas HDR picked up image HD#i.

Further, the standard luminance picked up image PL1 and the lowluminance picked up image PH1 picked up by the reference camera unit 21₁ and the HDR picked up image HD1 (obtained from the standard luminancepicked up image PL1 and the low luminance picked up image PH1) arehereinafter referred to each as reference image.

Further, the standard luminance picked up image PL#i and the lowluminance picked up image PH#i picked up by the peripheral camera unit21 _(i) and the HDR picked up image HD#i (obtained from the standardluminance picked up image PL#i and the low luminance picked up imagePH#i) are hereinafter referred to each as peripheral image.

The lens design data acquisition section 34 acquires lens design data ofthe emulation lens that is an optical lens of an emulation target andsupplies the acquired lens design data to the emulation lens informationgeneration section 37.

Here, the lens design data is stored, for example, in a memory notdepicted or is provided from a server or the like on the Internet. Thelens design data acquisition section 34 acquires and supplies the lensdesign data from the memory or the server on the Internet to theemulation lens information generation section 37.

It is to be noted that the emulation lens need not be an existingoptical lens but may be an optical lens that does not exist actually. Anoptical lens that does not exist actually may be an optical lens thatmay exist theoretically or may be an optical lens that may not existtheoretically.

Where an optical lens that does not exist is adopted as the emulationlens, the lens design data of the emulation lens is inputted, forexample, by a user operating an operation section not depicted. The lensdesign data acquisition section 34 acquires the lens design datainputted by the user.

The lens emulation section 35 performs a lens emulation process andsupplies an emulation image obtained by the lens emulation process tothe display apparatus 13 (FIG. 1).

In the lens emulation process, the lens emulation section 35 generatesan emulation image, which is an image that may be obtained if an imageof an image pickup object is picked up using the emulation lens, using aparallax map supplied from the parallax information generation section31 (as occasion demands, including a multilayer parallax map hereinafterdescribed), picked up images HD1 to HD7 of the seven visual pointssupplied from the saturated pixel restoration section 33 and lens designdata supplied from the lens design data acquisition section 34.

Accordingly, the lens emulation section 35 functions as an emulator thatperforms emulation of an image pickup apparatus (not depicted) having anemulation lens.

The lens emulation section 35 includes the incident ray reproductionsection 36, the emulation lens information generation section 37 and thelight condensing processing section 38.

The incident ray reproduction section 36 performs an incident rayreproduction process for reproducing (information of) rays incident to avirtual lens, which is a virtual optical lens, from a real space pointin a real space as an image process for the picked up images HD1 to HD7of the seven visual points using the picked up images HD1 to HD7 of theseven visual points supplied from the saturated pixel restorationsection 33 and a parallax map supplied from the parallax informationgeneration section 31.

Here, the virtual lens to which rays reproduced by the incident rayreproduction section 36 are incident is a virtual lens having asynthetic aperture provided by the camera units 21 ₁ to 21 ₇ as aplurality of image pickup sections for picking up picked up images HD1to HD7 (PL1 to PL7) of the seven visual points supplied to the incidentray reproduction section 36.

Where the camera units 21 ₁ to 21 ₇ are disposed, for example, in aregular hexagon as depicted in FIG. 2 and the distance between onecamera unit 21 _(i) and another camera unit 21 _(j) positioned nearestto the camera unit 21 _(i) is L, the synthetic aperture that is theaperture of the virtual lens has a substantially circular shape thatinterconnects the optical axes of the peripheral camera units 21 ₂ to 21₇ to each other and has a diameter of 2L.

The incident ray reproduction section 36 reproduces rays incident to thevirtual lens and supplies the rays to the light condensing processingsection 38.

The emulation lens information generation section 37 generates emulationlens information, which defines characteristics of the emulation lens,namely, defines rays that pass the emulation lens, using the parallaxmap supplied from the parallax information generation section 31 andlens design data supplied from the lens design data acquisition section34 and supplies the emulation lens information to the light condensingprocessing section 38.

Here, in the following description, the emulation lens information isalso referred to simply as lens information.

Since the lens information has a value equivalent to that of theemulation lens, it can be made a target of buying and selling. Since thelens information is electronic data and is easy to duplicate, in orderto prevent illegal duplication, it is possible to require certificationfor use of the lens information.

The light condensing processing section 38 performs a (digital) lightcondensing process for condensing, using the parallax map supplied fromthe parallax information generation section 31, rays supplied from theincident ray reproduction section 36 and lens information supplied fromthe emulation lens information generation section 37, the rays on thevirtual sensor, which is a virtual image sensor, through the emulationlens.

Then, the light condensing processing section 38 supplies an emulationimage obtained as a result of the light condensing process to thedisplay apparatus 13 (FIG. 1).

It is to be noted that it is possible to configure the image processingapparatus 12 as a server and also possible to configure the imageprocessing apparatus 12 as a client. Further, it is possible toconfigure the image processing apparatus 12 as a server-client system.Where the image processing apparatus 12 is configured as a server-clientsystem, it is possible to configure an arbitrary block or blocks of theimage processing apparatus 12 from a server and configure the remainingblocks from a client.

<Process of Image Processing System>

FIG. 4 is a flow chart illustrating an example of a process of the imageprocessing system of FIG. 1.

At step S1, the image pickup apparatus 11 picks up picked up images PL1to PL7 and PH1 to PH7 of the seven visual points as a plurality ofvisual points. The picked up images PL#i are supplied to the parallaxinformation generation section 31 and the saturated pixel restorationsection 33 of the image processing apparatus 12 (FIG. 3) and the pickedup images PH#i are supplied to the saturated pixel restoration section33.

Then, the processing advances from step S1 to step S2, at which theparallax information generation section 31 performs a parallaxinformation generation process for determining parallax informationusing the picked up images PL#i supplied from the image pickup apparatus11 and generating a parallax map (including a multilayer parallax map)in which the parallax information is registered.

The parallax information generation section 31 supplies the parallax mapobtained by the parallax information generation process to the incidentray reproduction section 36, the emulation lens information generationsection 37 and the light condensing processing section 38 that configurethe lens emulation section 35, and then the processing advances fromstep S2 to step S3.

At step S3, the saturated pixel restoration section 33 performs asaturated pixel restoration process for restoring the pixel value of asaturated pixel from among the pixels of the picked up image PL#isupplied from the camera unit 21 _(i) using the picked up image PH#isupplied from the camera unit 21 _(i).

The saturated pixel restoration section 33 supplies a picked up imageHD#i of a high dynamic range obtained by the saturated pixel restorationprocess to the incident ray reproduction section 36, and then, theprocessing advances from step S3 to step S4.

At step S4, the lens design data acquisition section 34 acquires lensdesign data of the emulation lens and supplies the lens design data tothe emulation lens information generation section 37.

Further, at step S4, the emulation lens information generation section37 performs an emulation lens information generation process forgenerating lens information of the emulation lens using the parallax mapsupplied from the parallax information generation section 31 and thelens design data supplied from the lens design data acquisition section34.

The emulation lens information generation section 37 supplies the lensinformation obtained by the emulation lens information generationprocess to the light condensing processing section 38, and then theprocessing advances from step S4 to step S5.

At step S5, the incident ray reproduction section 36 performs anincident ray reproduction process for reproducing rays to enter thevisual lens from a real space point in a rear space using the picked upimages HD1 to HD7 of the seven visual points supplied from the saturatedpixel restoration section 33 and the parallax map supplied from theparallax information generation section 31.

The incident ray reproduction section 36 supplies (the information of)the rays obtained by the incident ray reproduction process to the lightcondensing processing section 38, and then, the processing advances fromstep S5 to step S6.

At step S6, the light condensing processing section 38 performs a lightcondensing process for condensing the rays on the virtual sensor throughthe emulation lens using the parallax map supplied from the parallaxinformation generation section 31, rays supplied from the incident rayreproduction section 36 and lens information supplied from the emulationlens information generation section 37.

The light condensing processing section 38 supplies an emulation imageobtained as a result of the light condensing process to the displayapparatus 13, and then, the processing advances from step S6 to step S7.

At step S7, the display apparatus 13 displays the emulation image fromthe light condensing processing section 38.

<Generation of Parallax Map>

FIG. 5 is a block diagram depicting an example of a configuration of theparallax information generation section 31 of FIG. 3.

Referring to FIG. 5, the parallax information generation section 31includes a reference parallax map generation section 41 and a multilayerparallax map generation section 42.

To the reference parallax map generation section 41, picked up imagesPL1 to PL7 are supplied from the image pickup apparatus 11.

The reference parallax map generation section 41 generates a referenceparallax map, which is a parallax map in which parallaxes of thereference image PL1 that is one of the picked up images PL1 to PL7 fromthe image pickup apparatus 11 from the other picked up images(peripheral images) PL2 to PL7 are registered, and supplies thereference parallax map to the multilayer parallax map generation section42.

The multilayer parallax map generation section 42 uses, for example, thereference parallax map from the reference parallax map generationsection 41 to generate parallax maps of the peripheral images PL2 toPL7.

Then, the multilayer parallax map generation section 42 uses thereference parallax map of the reference image PL1 and the parallax mapsof the peripheral images PL2 to PL7 to generate a multilayer parallaxmap in which parallaxes with reference to the visual point (position) ofthe reference camera unit 21 ₁ are registered.

A necessary parallax map or maps from among the reference parallax mapof the reference image PL1, the parallax maps of the peripheral imagesPL2 to PL7 and the multilayer parallax map are supplied to the incidentray reproduction section 36, the emulation lens information generationsection 37 and the light condensing processing section 38 (FIG. 3).

FIG. 6 is a view illustrating an example of generation of a referenceparallax map by the reference parallax map generation section 41 of FIG.5.

In particular, FIG. 6 depicts an example of the picked up images PL1 toPL7.

In FIG. 6, in the picked up images PL1 to PL7, a predetermined objectobj is reflected as a foreground at the front side of a predeterminedbackground. Since the picked up images PL1 to PL7 are different invisual point from each other, the positions of the object obj reflected,for example, in the picked up images PL2 to PL7 are displaced bydistances corresponding to differences in visual point from the positionof the object obj reflected in the reference image PL1.

The reference parallax map generation section 41 successively selectsthe pixels of the reference image PL1 as a noticed pixel and detects acorresponding pixel (corresponding point) corresponding to the noticedpixel from within each of the other picked up images PL2 to PL7, namely,from within each of the peripheral images PL2 to PL7.

As a method for detecting a corresponding pixel of each of theperipheral images PL2 to PL7 corresponding to the noticed pixel of thereference image PL1, an arbitrary method such as, for example, blockmatching can be adopted.

Here, a vector heading from the noticed pixel of the reference image PL1toward a corresponding pixel of a peripheral image PL#i, namely, avector representative of a positional displacement between the noticedpixel and the corresponding pixel, is referred to as parallax vectorv#i,1.

The reference parallax map generation section 41 determines parallaxvectors v2,1 to v7,1 of the respective peripheral images PL2 to PL7.Then, the reference parallax map generation section 41 performs majorityvote on magnitude of the parallax vectors v2,1 to v7,1 and determinesthe magnitude of the parallax vector v#i,1 that wins in the majorityvote as a parallax of the (position of) the noticed pixel.

Here, where the distances between the reference camera unit 21 ₁ thatpicks up the reference image PL1 and the peripheral camera units 21 ₂ to21 ₇ that pick up the peripheral images PL2 to PL7 are the equaldistance L in the image pickup apparatus 11 as described hereinabovewith reference to FIG. 2, if a portion reflected at the noticed pixel ofthe reference image PL1 is reflected also in the peripheral images PL2to PL7, then vectors having an equal magnitude although the directionsare different from each other are determined as the parallax vectorsv2,1 to v7,1.

In particular, in this case, the parallax vectors v2,1 to v7,1 arevectors that have an equal magnitude but have directions according tothe positions (visual points) of the peripheral images PL2 to PL7 withrespect to the reference camera unit 21 ₁.

However, since the picked up images PL1 to PL7 have visual pointsdifferent from each other, the peripheral images PL2 to PL7 possiblyinclude an image that suffers from occlusion, namely, in which a portionreflected at the noticed pixel of the reference image PL1 is hidden bythe foreground and is not reflected.

In regard to the peripheral image (hereinafter referred to also asocclusion image) PL#i in which a portion reflected at the noticed pixelof the reference image PL1 is not reflected, it is difficult to detect acorrect pixel as the corresponding pixel that corresponds to the noticedpixel.

Therefore, as regards the occlusion image PL#i, a parallax vector v#i,1having a magnitude different from that of a parallax vector v#j,1 of aperipheral image PL#j in which a portion reflected at the noticed pixelof the reference image PL1 is reflected is determined.

It is estimated that the number of images that suffers from occlusion inregard to the noticed pixel is smaller than that of images that suffersfrom occlusion among the peripheral images PL2 to PL7. Therefore, thereference parallax map generation section 41 performs majority vote onmagnitude of the parallax vectors v2,1 to v7,1 as described above anddetermines the magnitude of the parallax vector v#i,1 that wins in themajority vote as a parallax of the noticed pixel.

In FIG. 6, the three parallax vectors v2,1, v3,1 and v7,1 are vectorshaving an equal magnitude among the parallax vectors v2,1 to v7,1.Meanwhile, in the reference vectors v4,1, v5,1 and v6,1, parallaxvectors having an equal magnitude do not exist.

Therefore, the magnitude of the three parallax vectors v2,1, v3,1 andv7,1 is determined as a parallax of the noticed pixel.

It is to be noted that the direction of the parallax of the noticedpixel of the reference image PL1 from an arbitrary peripheral image PL#ican be recognized from a positional relationship between the referencecamera unit 21 ₁ and the peripheral camera unit 21 _(i).

The reference parallax map generation section 41 successively selectsthe pixels of the reference image PL1 as a noticed pixel and determinesthe parallax. Then, the reference parallax map generation section 41generates a parallax map in which the parallax of each pixel of thereference image PL1 in association with the position (xy coordinates) ofthe pixel is registered as a reference parallax map. Accordingly, theparallax map is a map (table) in which positions of pixels andparallaxes of the pixels are associated with each other.

Here, in addition to the parallax map of the reference image PL1(reference parallax map), also the parallax map of each peripheral imagePL#i can be generated similarly.

However, in generation of the parallax map of the peripheral image PL#i,the majority vote of parallax vectors is performed with the magnitude ofthe parallax vector adjusted on the basis of a relationship in visualpoint between the peripheral image PL#i and each of the other picked upimages PL#j (positional relationship between the camera units 21 _(i)and 21 _(j)).

In particular, for example, where the parallax map of the peripheralimage PL5 is to be generated, the parallax vector obtained, for example,between the peripheral image PL5 and the reference image PL1 has amagnitude equal to twice the parallax vector obtained between theperipheral image PL5 and the peripheral image PL2.

This is because, while the baseline length that is a distance betweenthe optical axes of the peripheral camera unit 21 ₅ that picks up theperipheral image PL5 and the reference camera unit 21 ₁ that picks upthe reference image PL1 is the distance L, the baseline length betweenthe peripheral camera unit 21 ₅ that picks up the peripheral image PL5and the peripheral camera unit 21 ₂ that picks up the peripheral imagePL2 is the distance 2L.

Therefore, if it is assumed that, for example, the distance L that isthe baseline length between the peripheral camera unit 21 ₅ and thereference camera unit 21 ₁ is called reference baseline length, then themajority vote on parallax vector is performed after the magnitude of theparallax vectors is adjusted such that the baseline length is convertedinto the reference baseline length L.

In particular, for example, since the baseline length L, for example,between the peripheral camera unit 21 ₅ that picks up the peripheralimage PL5 and the reference camera unit 21 ₁ that picks up the referenceimage PL1 is equal to the reference baseline length L, the parallaxvector obtained between the peripheral image PL5 and the reference imagePL1 is adjusted in magnitude to one time.

Meanwhile, since the baseline length 2L, for example, between theperipheral camera unit 21 ₅ that picks up the peripheral image PL5 andthe peripheral camera unit 21 ₂ that picks up the peripheral image PL2is equal to twice the reference baseline length L, the parallax vectorobtained between the peripheral image PL5 and the reference image PL1 isadjusted in magnitude to ½ time (n times where n is a value of the ratioof the baseline length between the peripheral camera unit 21 ₅ and theperipheral camera unit 21 ₂ to the reference baseline length).

Also the parallax vector obtained between the peripheral image PL5 andany other picked up image PL#i is adjusted in magnitude to n times wheren is a value of the ratio to the reference baseline length L similarly.

Then, the majority vote on parallax vector is performed using theparallax vectors after the adjustment in magnitude.

It is to be noted that the reference parallax map generation section 41can determine the parallax of (each of the pixels of) the referenceimage PL1, for example, with the accuracy of a pixel of a picked upimage picked up by the image pickup apparatus 11. Further, the parallaxof the reference image PL1 can be determined, for example, with a fineraccuracy than a pixel of a picked up image picked up by the image pickupapparatus 11 (hereinafter referred to as subpixel accuracy), inparticular, for example, with an accuracy of a ¼ pixel or the like.

Where a parallax is determined with a subpixel accuracy, in a process inwhich a parallax is used, not only it is possible to use the parallax ofthe subpixel accuracy as it is but also it is possible to use theparallax by integrating the parallax of the subpixel accuracy byrounding down, rounding up or rounding off decimal places of theparallax.

In the present embodiment, the parallax is determined with a subpixelaccuracy and, unless otherwise specified, the parallax with the subpixelaccuracy is integrated and used in order to facilitate calculation.

FIG. 7 is a view illustrating an example of generation of a parallax mapof a peripheral image PL#i.

The parallax map of the peripheral image PL#i not only can be generatedsimilarly to the parallax map of the reference image PL1 (referenceparallax map) but also can be generated, as it were, simply and easilyutilizing the reference parallax map.

The multilayer parallax map generation section 42 (FIG. 5) can generatethe parallax map of the peripheral image PL#i utilizing the referenceparallax map.

In FIG. 7, the parallax maps of the peripheral images PL2 and PL5 aregenerated utilizing the reference parallax map.

Here, in FIG. 7, the reference image PL1 and the peripheral images PL2and PL5 and the parallax maps of the reference image PL1 and theperipheral images PL2 and PL5 are depicted.

As the parallax map of the reference image PL1 (reference parallax map),a plan view of a parallax map in which the parallaxes of the pixels arerepresented by shading and a parallax map in which the axis of abscissaindicates the horizontal position of a pixel and the axis of ordinateindicates the parallax are depicted.

This similarly applies also to the parallax maps of the peripheralimages PL2 and PL5.

When the multilayer parallax map generation section 42 is to utilize thereference parallax map to generate a parallax map of a peripheral imagePL#i, it moves the parallaxes registered at the positions of pixels bythe parallaxes in directions according to the positional relationshipbetween the camera unit 21 ₁ that picks up the reference image and theperipheral camera unit 21 _(i) that picks up the peripheral image PL#i(the direction is hereinafter referred to as camera position relationdirection) in the reference parallax map to generate the parallax map ofthe peripheral image PL#i.

For example, when the parallax map of the peripheral image PL2 is to begenerated, determining the leftward direction, which is a direction whenthe camera unit 21 ₁ that picks up the reference image is viewed fromthe camera unit 21 ₂ that picks up the peripheral image PL2 as thecamera position relation direction, the parallax registered at theposition of each pixel of the reference parallax map is moved by theparallax in the leftward direction that is the camera position relationdirection to generate the parallax map of the peripheral image PL2.

On the other hand, for example, when the parallax map of the peripheralimage PL5 is to be generated, determining the rightward direction, whichis a direction when the camera unit 21 ₁ that picks up the referenceimage is viewed from the camera unit 21 ₅ that picks up the peripheralimage PL5 as the camera position relation direction, the parallaxregistered at the position of each pixel of the reference parallax mapis moved by the parallax in the rightward direction that is the cameraposition relation direction to generate the parallax map of theperipheral image PL5.

When the parallax map of the peripheral image PL#i is generatedutilizing the reference parallax map in such a manner as describedabove, in the parallax map of the peripheral image PL#i, an areacorresponding to pixels in a region that is not reflected in thereference image P1 although it is reflected in the peripheral image PL#iis a non-registration area (portion indicated by slanting lines in FIG.7) in which no parallax is registered.

Therefore, the multilayer parallax map generation section 42interpolates parallaxes in a non-registration area of the parallax mapof the peripheral image PL#i generated utilizing the reference parallaxmap to complete the parallax map of the peripheral image PL#i.

FIG. 8 is a view illustrating interpolation of a parallax into anon-registration area of the parallax map of the peripheral image PL#i.

Here, also in FIG. 8, the reference image PL1 and the peripheral imagesPL2 and PL5 and the parallax maps of the reference image PL1 and theperipheral images PL2 and PL5 are depicted similarly as in FIG. 7.

The multilayer parallax map generation section 42 follows a straightline of the camera position relation direction, which is a straight lineextending in the camera position relation direction from a pixel in anon-registration area in the parallax map of the peripheral image PL#i,in both of one direction and the opposite direction and detects aparallax registration pixel that is reached first in the followingprocess and is a pixel whose parallax is registered.

Further, the multilayer parallax map generation section 42 selects asmaller parallax (parallax corresponding to a greater distance) frombetween the parallax of the parallax registration pixel in the onedirection and the parallax of the parallax registration pixel in theopposite direction of the camera position relation direction straightline as an interpolation parallax to be used for interpolation of apixel in the non-registration area.

Then, the multilayer parallax map generation section 42 interpolates theparallax of the pixel in the non-registration area with theinterpolation parallax (registers the interpolation parallax as theparallax of the pixel in the non-registration area) to complete theparallax map of the peripheral image PL#i.

In FIG. 8, in the parallax map of the peripheral image PL2, a parallaxregistered at a pixel (parallax registration pixel) neighboring with theboundary at the right side of the non-registration area (FIG. 7) isselected as an interpolation parallax, and parallaxes of pixels in thenon-registration area are interpolated as interpolation parallaxes(interpolation parallaxes are propagated as parallaxes of pixels in thenon-registration area).

Further, in FIG. 8, in the parallax map of the peripheral image PL5, aparallax registered at a pixel (parallax registration pixel) neighboringwith the boundary at the left side of the non-registration area (FIG. 7)is selected as an interpolation parallax, and parallaxes of pixels inthe non-registration area are interpolated as interpolation parallaxes.

FIG. 9 is a view illustrating an example of generation of a multilayerparallax map.

Here, also in FIG. 9, the reference image PL1 and the peripheral imagesPL2 and PL5 and the parallax maps of the reference image PL1 and theperipheral images PL2 and PL5 are depicted similarly as in FIGS. 7 and8.

The multilayer parallax map generation section 42 uses the referenceparallax map of the reference image PL1 and (one or more of) theparallax maps of the peripheral images PL2 to PL7 to generate amultilayer parallax map.

In particular, the multilayer parallax map generation section 42successively selects the pixels of the peripheral image PL#i as anoticed pixel and detects a corresponding pixel of the reference imagecorresponding to the noticed pixel.

For example, a pixel of the reference image PL1 at a position moved inthe camera position relation direction (here, in the direction in whichthe camera unit 22 _(i) is viewed from the camera unit 21 ₁) by theparallax registered at the noticed pixel of the parallax map of theperipheral image PL#i from the position of the noticed pixel of theperipheral image PL#i is detected as a corresponding pixel of thereference image PL1 corresponding to the noticed pixel of the peripheralimage PL#i.

Then, the parallax of the noticed pixel of the peripheral image PL#i isregistered into the corresponding pixel of the reference image PL1 inthe reference parallax map.

In generation of a multilayer parallax map, although a parallax isregistered already at the corresponding pixel of the reference image PL1in the reference parallax map, where the parallax of the noticed pixelof a peripheral image PL#i is different from the parallax registeredalready at the corresponding pixel, it is registered in such a form thatit is added to the parallax registered already.

As described above, a parallax registered in the parallax map of theperipheral image PL#i is, as it were, reflected in the referenceparallax map in such a form that it is added, and the reference parallaxmap after the reflection is the multilayer parallax map.

As a result, the multilayer parallax map is a parallax map in which, inaddition to parallaxes in a region that can be viewed from the visualpoint of the reference camera unit 21 ₁ (hereinafter referred tosometimes as reference visual point), parallaxes at least at a portionof a region that cannot be viewed hiding behind the foreground (regionin which occlusion occurs) are registered.

In the multilayer parallax map, for example, at a pixel in the region ofthe foreground, as it were, multilayer parallaxes (a plurality ofparallaxes) like parallaxes corresponding to distances to the foregroundand parallaxes corresponding to distances to the background that cannotbe viewed hiding behind the foreground from the reference visual pointare registered.

FIG. 10 is a flow chart illustrating an example of a process forgeneration of a reference parallax map and a multilayer parallax map bythe parallax information generation section 31 of FIG. 5.

In the reference parallax map generation process for generating areference parallax map, at step S11, the reference parallax mapgeneration section 41 selects one of images, which have not beenselected as a noticed image as yet, from among the peripheral images PL2to PL7 from the image pickup apparatus 11 as a noticed image.Thereafter, the processing advances to step S12.

At step S12, the reference parallax map generation section 41 detects aparallax vector v (FIG. 6) between the noticed image and each pixel ofthe reference image PL1 from the image pickup apparatus 11. Thereafter,the processing advances to step S13.

At step S13, the reference parallax map generation section 41 decideswhether or not all of the peripheral images PL2 to PL7 have beenselected as a noticed image.

If it is decided at step S13 that all of the peripheral images PL2 toPL7 have not been selected as a noticed image as yet, then theprocessing returns to step S11, and thereafter, similar processes arerepeated.

On the other hand, if it is decided at step S13 that all of theperipheral images PL2 to PL7 have been selected as a noticed image, thenthe processing advances to step S14.

At step S14, the reference parallax map generation section 41 performsmajority vote on magnitude of the parallax vectors v2,1 to v7,1 of theperipheral images PL2 to PL7 in regard to each pixel of the referenceimage as described hereinabove with reference to FIG. 6 and determinesthe magnitude of the parallax vector v#i,1 that wins the majority voteas a parallax.

Then, the reference parallax map generation section 41 generates areference parallax map in which the parallaxes are registered for eachpixel of the reference image and supplies the reference parallax map tothe multilayer parallax map generation section 42, thereby ending thereference parallax map generation process.

In the multilayer parallax map generation process for generating amultilayer parallax map, at step S21, the multilayer parallax mapgeneration section 42 generates parallax maps of the peripheral imagesPL2 to PL7 using the reference parallax map from the reference parallaxmap generation section 41 as described hereinabove with reference toFIG. 7. Thereafter, the processing advances to step S22.

At step S22, the multilayer parallax map generation section 42interpolates parallaxes into non-registration areas of the parallax mapsof the peripheral images PL#i as described hereinabove with reference toFIG. 8 to complete the parallax maps of the peripheral images PL#i.Thereafter, the processing advances to step S23.

At step S23, the multilayer parallax map generation section 42 reflectsthe parallax maps of the peripheral images PL2 to PL7 on the referenceparallax map to generate a multilayer parallax map as describedhereinabove with reference to FIG. 9, thereby ending the multilayerparallax map generation process.

<Restoration of Saturated Pixel>

FIG. 11 is a view schematically depicting an example of an actual imageobtained by image pickup of a predetermined image pickup object using anactual optical lens.

A of FIG. 11 depicts an example of an actual image when the focus is setto pan focus.

In the actual image of A of FIG. 11, a bulb positioned at the back sideis reflected comparatively clearly without a blur.

B of FIG. 11 depicts an example of an actual image when the focus is setto a comparatively near position, for example, to a distance of 1 m(from the principal point of the optical lens).

In the actual image of B of FIG. 11, although the image pickup object(in FIG. 11, a can) at the distance of 1 m is reflected clearly withouta blur, other image pickup objects at different distances are reflectedblurred. Further, while, in the actual image of B of FIG. 11, a bulbpositioned at the back side is blurred, since it is high in luminance,it is reflected comparatively clearly.

FIG. 12 is a view depicting an example of an emulation image obtained bythe image processing apparatus 12 of FIG. 3.

A of FIG. 12 depicts an example of an emulation image obtained when theimage processing apparatus 12 does not perform a saturated pixelrestoration process.

In the emulation image of A of FIG. 12, the focus is set to a positionat the near side similarly as in the case of B of FIG. 11, andtherefore, the bulb positioned at the back side is blurred.

However, although, in the emulation image of A of FIG. 12, the bulbpositioned at the back side is blurred, different from the case of B ofFIG. 11, the bulb is not very clear.

B of FIG. 12 depicts an example of an emulation image obtained when theimage processing apparatus 12 performs a saturated pixel restorationprocess.

In the emulation image of B of FIG. 12, the focus is set to a positionat the near side similarly as in the case of B of FIG. 11, andtherefore, the bulb positioned at the back side is blurred.

Furthermore, in the emulation image of B of FIG. 12, the bulb positionedat the back side is blurred clearly similarly as in the case of B ofFIG. 11.

Accordingly, according to the saturated pixel restoration process, ablur degree of an actual optical lens can be reproduced accurately by anemulation process performed later.

In particular, according to the saturated pixel restoration process, aclear blur similar to that of an actual image picked up using an actualoptical lens can be reproduced.

FIG. 13 is a view illustrating a principle by which a clear blur is notreproduced when the saturated pixel restoration process is notperformed.

A of FIG. 13 depicts an example of light intensity of an image pickupobject.

In A of FIG. 13, the axis of abscissa indicates a position in thehorizontal direction (horizontal coordinate) of an image sensor notdepicted from which the camera unit 21 _(i) is configured, and the axisof ordinate indicates the light intensity of light from an image pickupobject irradiated on the image sensor.

In A of FIG. 13, light of a very high light intensity S₀ irradiates (theimage sensor of) the camera unit 21 _(i).

B of FIG. 13 depicts an example of the luminance of a picked up imageoutputted from the camera unit 21 _(i) when light of the light intensityS₀ is received.

In B of FIG. 13, the axis of abscissa represents the position of a pixelin the horizontal direction of a picked up image outputted from thecamera unit 21 _(i), which receives light of the light intensity S₀, andthe axis of ordinate represents the luminance of a pixel of the pickedup image.

The luminance corresponding to light of the light intensity S₀ exceedsan image pickup limit luminance THL that is a maximum value that can beoutputted as a pixel value from the camera unit 21 _(i), and therefore,in the picked up image, the luminance of an image pickup object fromwhich light of the light intensity S₀ is emitted is cut (clamped) to theimage pickup limit luminance THL.

Here, the light intensity corresponding to the image pickup limitluminance THL is represented as S₁ (<S₀).

C of FIG. 13 depicts an example of an emulation image generated by alens emulation process in which a picked up image whose luminancecorresponding to light of the light intensity S₀ is cut to the imagepickup limit luminance THL corresponding to the light intensity S₁ isused.

In C of FIG. 13, the axis of abscissa indicates the position of a pixelin the horizontal direction of an emulation image, and the axis ofordinate indicates the luminance of the pixel of the emulation image.

When, in generation of an emulation image, an image pickup objectreflected on pixels having a pixel value equal to the image pickup limitluminance THL corresponding to the light intensity S₁ is blurred, thelight intensity S₁ is spread around the pixels on which the image pickupobject is reflected, and the luminance of the image pickup objectfurther drops from the image pickup limit luminance THL.

As described above, an image pickup object that emits light of the lightintensity S₀ higher than the light intensity S₁ corresponding to theimage pickup limit luminance THL (for example, a bulb or the like) isreflected as an image pickup object that emits light of the lightintensity S₁ corresponding to the image pickup limit luminance THL in apicked up image.

Then, if, in generation of an emulation image in which a picked up imagein which an image pickup object that emits light of the light intensityS₁ corresponding to the image pickup limit luminance THL is reflected isused, the image pickup object is blurred, then the light intensity S₁lower than the original light intensity S₀ is spread and clearness doesnot appear on the image pickup object.

FIG. 14 is a view illustrating a principle by which a clear blur isreproduced by performing the saturated pixel restoration process.

A of FIG. 14 depicts an example of the luminance of a picked up imageoutputted when light of the light intensity S₀ is received by the cameraunit 21 _(i).

The luminance of the picked up image of A of FIG. 14 is similar to thatof B of FIG. 13, and the luminance of an image pickup object that emitslight of the light intensity S₀ is cut to the image pickup limitluminance THL corresponding to the light intensity S₁ that is lower thanthe light intensity S₀.

B of FIG. 14 depicts an example of the luminance of a picked up imageafter the saturated pixel restoration process.

In the saturated pixel restoration process, as a pixel value of an imagepickup object having a pixel value cut to the image pickup limitluminance THL in a picked up image, a luminance obtained by adding aluminance corresponding to a light intensity S₂ that satisfies anexpression S₁+S₂≈S₀ to the image pickup limit luminance THL is restored.

As a result, in the picked up image after the saturated pixelrestoration process, a pixel whose pixel value is cut to the imagepickup limit luminance THL has a luminance substantially correspondingto the original light intensity S₀≈S₁+S₂ as a pixel value.

C of FIG. 14 depicts an example of an emulation image generated using apicked up image after the saturated pixel restoration process.

In C of FIG. 14, the axis of abscissa represents a position of a pixelin the horizontal direction of an emulation image, and the axis ofordinate represents the luminance of the pixel of the emulation image.

When, in generation of an emulation image, an image pickup objectreflected on a pixel having a luminance corresponding to the lightintensity S₁+S₂ as a pixel value is blurred, although the lightintensity S₁+S₂ is spread around the pixel on which the image pickupobject is reflected, since the light intensity S₁+S₂ is very high, evenif the light intensity S₁+S₂ is spread, the luminance of the imagepickup object is higher than that when the saturated pixel restorationprocess is not performed.

As described above, according to the saturated pixel restorationprocess, the luminance of an image pickup object (for example, a bulb orthe like) that emits light of the high light intensity S₀ and whosepixel value is cut to the image pickup limit luminance THL is restoredto a luminance corresponding to the (substantially) original high lightintensity S₀.

Then, when, in generation (lens emulation process) of an emulation imagein which a picked up image in which an image pickup object of aluminance corresponding to such a high light intensity S₀ as describedabove is reflected is used, the image pickup object is blurred, a clearblur appears on the image pickup object.

Accordingly, a clear blur similar to that in an actual image picked upusing an actual optical lens can be reproduced.

Here, the saturated pixel restoration section 33 performs a saturatedpixel restoration process by which a pixel value of a saturated pixelwhose pixel value is saturated from among pixels of the standardluminance picked up image PL#i supplied from the camera unit 21 _(i) isrestored using the low luminance picked up image PH#i supplied from thecamera unit 21 _(i) as described hereinabove with reference to FIG. 3.

That the pixel value of a pixel of the standard luminance picked upimage PL#i is saturated signifies that the pixel value of the pixel ofthe standard luminance picked up image PL#i is cut to the image pickuplimit luminance THL or the pixel value of the pixel of the standardluminance picked up image PL#i is proximate to the image pickup limitluminance THL as described hereinabove with reference to FIGS. 13 and14.

Further, in the saturated pixel restoration process, a standardluminance picked up image PL#i and a low luminance picked up image PH#iin both of which the same image pickup object is reflected are required.As a method for acquiring the standard luminance picked up image PL#iand the low luminance picked up image PH#i in both of which the sameimage pickup object is reflected, an arbitrary method can be adopted.

In the following, as the method for acquiring a standard luminancepicked up image PL#i and a low luminance picked up image PH#i in both ofwhich the same image pickup object is reflected, a first acquisitionmethod, a second acquisition method and a third acquisition method aredescribed.

It is to be noted that a picked up image that is a target of thesaturated pixel restoration process is an image of RAW data orpreferably is a demosaic image before gamma correction in order to keepthe linearity of the luminance.

FIG. 15 is a view illustrating the first acquisition method foracquiring a standard luminance picked up image PL#i and a low luminancepicked up image PH#i.

In the first acquisition method, a plural number of times such as twotimes of image pickup are performed in a short period of time while theexposure time period (shutter speed) is changed for all of the cameraunits 21 ₁ to 21 ₇ configuring the image pickup apparatus 11.

In particular, in the first acquisition method, image pickup in astandard exposure time period (exposure time period estimated to beappropriate upon image pickup, for example, set by an automatic exposurefunction or the like) and image pickup in an exposure time periodshorter than the standard exposure time period are successivelyperformed by all of the camera units 21 ₁ to 21 ₇ that configure theimage pickup apparatus 11.

Each picked up image picked up in the standard exposure time period is astandard luminance picked up image PL#i, and each picked up image pickedup in the exposure time period shorter than the standard exposure timeperiod is a low luminance picked up image PH#i.

Accordingly, the standard luminance picked up image PL#i and the lowluminance picked up image PH#i obtained by the first acquisition methodare images picked up at timings different from each other.

Now, if the standard exposure time period is represented as X (seconds),then as the exposure time period of the low luminance picked up imagePH#i shorter than the standard exposure time period X, for example, X/16(second) or the like can be adopted.

FIG. 15 depicts an example of the standard luminance picked up imagesPL1, PL2 and PL5 and the low luminance picked up images PH1, PH2 and PH5picked up by the camera units 21 ₁, 21 ₂ and 21 ₅, respectively.

Since the exposure time period of the low luminance picked up image PH#iis shorter than the standard exposure time period of the standardluminance picked up image PL#i, an image pickup object in the lowluminance picked up image PH#i is reflected darker than that in thestandard luminance picked up image PL#i.

FIG. 16 is a block diagram depicting a first configuration example ofthe saturated pixel restoration section 33 of FIG. 3.

In particular, FIG. 16 depicts an example of a configuration of thesaturated pixel restoration section 33 where the standard luminancepicked up image PL#i and the low luminance picked up image PH#i areacquired by the first acquisition method.

Referring to FIG. 16, the saturated pixel restoration section 33includes a saturation decision section 51 and a restoration section 52.

To the saturation decision section 51, standard luminance picked upimages PL#i are supplied from the image pickup apparatus 11 (FIG. 1).

The saturation decision section 51 performs a saturation decision fordeciding whether or not each of the pixels of the standard luminancepicked up image PL#i from the image pickup apparatus 11 is a saturatedpixel and supplies a decision result of the saturation decision to therestoration section 52.

To the restoration section 52, in addition to a decision result of thesaturation decision from the saturation decision section 51, standardluminance picked up images PL#i and low luminance picked up images PH#iare supplied from the image pickup apparatus 11.

The restoration section 52 specifies saturated pixels from among thepixels of the standard luminance picked up images PL#i from the imagepickup apparatus 11 in response to the decision results (saturationdecision results) of the saturation decision from the saturationdecision section 51.

Further, the restoration section 52 restores the pixel value of eachsaturated pixel using the low luminance picked up images PH#i from theimage pickup apparatus 11 as occasion demands and supplies HDR picked upimages HD#i of a higher dynamic range than that of the standardluminance picked up images PL#i to the incident ray reproduction section36 (FIG. 3), the HDR picked up images HD#i being obtained by therestoration.

FIG. 17 is a flow chart illustrating an example of the saturated pixelrestoration process performed by the saturated pixel restoration section33 of FIG. 16.

At step S31, the saturation decision section 51 selects, from among thevisual points (positions) of the seven camera units 21 ₁ to 21 ₇, onevisual point that has not been selected as a noticed visual point as yetas a noticed visual point. Thereafter, the processing advances to stepS32.

At step S32, the saturation decision section 51 selects, from among thepixels of the standard luminance picked up image PL#i of the noticedvisual point from among the standard luminance picked up images PL1 toPL7 of the seven visual points supplied from the image pickup apparatus11, one pixel that has not been selected as a noticed pixel as yet as anoticed pixel. Thereafter, the processing advances to step S33.

At step S33, the saturation decision section 51 performs a saturationdecision of whether or not the standard pixel value that is a pixelvalue of the noticed pixel of the standard luminance picked up imagePL#i of the noticed visual point is equal to or higher than apredetermined threshold value TH1.

If it is decided at step S33 that the standard pixel value of thenoticed pixel of the standard luminance picked up image PL#i of thenoticed visual point is not equal to or higher than the threshold valueTH1, namely, if the standard pixel value of the noticed pixel of thestandard luminance picked up image PL#i of the noticed visual point isnot in a saturated state, then the saturation decision section 51supplies a saturation decision result that the standard pixel value isnot in a saturated state to the restoration section 52. Then, theprocessing advances to step S34.

At step S34, the restoration section 52 selects, in response to thesaturation decision result that the standard pixel value is not in asaturated state from the saturation decision section 51, the standardpixel value of the noticed pixel of the standard luminance picked upimage PL#i of the noticed visual point from the image pickup apparatus11 as a pixel value of a pixel at the position of the noticed pixel inthe HDR picked up image HD#i of the noticed visual point (also the pixelat the position of the noticed pixel is hereinafter referred to asnoticed pixel). Thereafter, the processing advances to step S37.

On the other hand, if it is decided at step S33 that the standard pixelvalue of the noticed pixel of the standard luminance picked up imagePL#i of the noticed visual point is equal to or higher than thethreshold value TH1, namely, if the standard pixel value of the noticedpixel of the standard luminance picked up image PL#i of the noticedvisual point is in a saturated state or the possibility of suchsaturation is high, then the saturation decision section 51 supplies thesaturation decision result that the standard pixel value is in asaturated state to the restoration section 52. Then, the processingadvances to step S35.

At step S35, the restoration section 52 decides, in response to thesaturation decision result that the standard pixel value is in asaturated state from the saturation decision section 51, whether or notthe low luminance pixel value that is a pixel value of a pixel at theposition of the noticed pixel of the low luminance picked up image PH#iof the noticed visual point from the image pickup apparatus 11 (also thepixel at the position of the noticed pixel is referred to as noticedpixel) is equal to or higher than a predetermined threshold value TH2that is lower than the threshold value TH1.

If it is decided at step S35 that the low luminance pixel value of thenoticed pixel of the low luminance picked up image PH#i of the noticedvisual point is not equal to or higher than the threshold value TH2,namely, if the low luminance pixel value of the noticed pixel of the lowluminance picked up image PH#i of the noticed visual point is low andthe possibility that the low luminance pixel value may be noise is high,then the processing advances to step S34.

At step S34, the restoration section 52 selects the standard pixel valueof the noticed pixel of the standard luminance picked up image PL#i ofthe noticed visual point as a pixel value of the noticed pixel of theHDR picked up image HD#i of the noticed visual point as described above.Then, the processing advances to step S37.

On the other hand, if it is decided at step S35 that the low luminancepixel vale of the noticed pixel of the low luminance picked up imagePH#i of the noticed visual point is equal to or higher than thethreshold value TH2, namely, if the low luminance pixel value of thenoticed pixel of the low luminance picked up image PH#i of the noticedvisual point is a value of a certain level with which it can be regardednot as noise, the processing advances to step S36.

At step S36, the restoration section 52 multiplies the low luminancepixel value of the noticed pixel of the low luminance picked up imagePH#i of the noticed visual point by a predetermined number anddetermines a resulting pixel value as a restored pixel value restoredfrom the saturated pixel. Further, the restoration section 52 selectsthe restored pixel value as a pixel value of the noticed pixel of theHDR picked up image HD#i of the noticed visual point. Then, theprocessing advances from step S36 to step S37.

Here, as the predetermined number for multiplication (hereinafterreferred to as restoration gain) that is used when a restored pixelvalue is to be determined, the value of the ratio between the exposuretime period of the standard luminance picked up image PL#i (standardexposure time period) and the exposure time period of the low luminancepicked up image PH#i is adopted.

Accordingly, for example, as described hereinabove with reference toFIG. 15, where the standard exposure time period is X (seconds) and theexposure time period of the low luminance picked up image PH#i is X/16(seconds), the restoration gain is 16=X/(X/16) times.

At step S37, the saturation decision section 51 decides whether or notall of the pixels of the standard luminance picked up image PL#i of thenoticed visual point have been selected as a noticed pixel.

If it is decided at step S37 that all of the pixels of the standardluminance picked up image PL#i of the noticed visual point have not beenselected as a noticed pixel, then the processing returns to step S32 andthen similar processes are repeated.

On the other hand, if it is decided at step S37 that all of the pixelsof the standard luminance picked up image PL#i of the noticed visualpoint have been selected as a noticed pixel, then the processingadvances to step S38.

At step S38, the saturation decision section 51 decides whether or notall of the seven visual points have been selected as a noticed visualpoint.

If it is decided at step S38 that all of the seven visual points havenot been selected as a noticed visual point, then the processing returnsto step S31, and thereafter, similar processes are repeated.

On the other hand, if it is decided at step S38 that all of the sevenvisual points have been selected as a noticed visual point, then therestoration section 52 supplies the HDR picked up images HD1 to HD7 ofthe seven visual points obtained by the processes described above to theincident ray reproduction section 36 (FIG. 3), thereby ending thesaturated pixel restoration process.

It is to be noted that, as the low luminance picked up image PH#i, afirst low luminance picked up image whose exposure time period isshorter than the standard exposure time period and a second lowluminance picked up image whose exposure time period is shorter thanthat of the first low luminance picked up image can be adopted.

In the first acquisition method, the image pickup apparatus 11 canacquire a standard luminance picked up image PL#i, a first low luminancepicked up image and a second low luminance picked up image bysuccessively performing image pickup by three times changing theexposure time period.

The second low luminance picked up image can be used to restore, where apixel value of a standard luminance picked up image is in a saturatedstate and also a pixel value of a first low luminance picked up image isin a saturated state, the saturated pixel whose pixel value is in asaturated state.

FIG. 18 is a block diagram depicting a second configuration example ofthe saturated pixel restoration section 33.

In particular, FIG. 18 depicts a configuration example of the saturatedpixel restoration section 33 where a standard luminance picked up imagePL#i and a low luminance picked up image PH#i in both of which the sameimage pickup object is reflected are acquired by the second acquisitionmethod.

It is to be noted that, in FIG. 18, like elements to those of FIG. 16are denoted by like reference numerals, and in the followingdescription, description of them is suitably omitted.

In the second acquisition method, all of the camera units 21 ₁ to 21 ₇configuring the image pickup apparatus 11 perform high speed imagepickup by which a plurality of times of image pickup are performed in ashort period of time with a fixed exposure time period shorter than astandard exposure time period.

Further, in the second acquisition method, from among a plurality ofhigh speed picked up images obtained by high speed image pickup of thecamera unit 21 _(i), different numbers of high speed picked up imagesare added to generate (acquire) a standard luminance picked up imagePL#i and a low luminance picked up image PH#i.

For example, where the exposure time period of high speed image pickupis 1/4000 second, if the exposure time period (standard exposure timeperiod) of the standard luminance picked up image PL#i is set to 1/60second and the exposure time period of the low luminance picked up imagePH#i is set to 1/1000 second, then the standard luminance picked upimage PL#i and the low luminance picked up image PH#i can be generatedin the following manner.

In particular, the standard luminance picked up image PL#i can begenerated by adding 66 or 67 high speed picked up images. Meanwhile, thelow luminance picked up image PH#i can be generated by adding four highspeed picked up images.

In FIG. 18, assuming that high speed picked up images are supplied fromthe image pickup apparatus 11, the saturated pixel restoration section33 acquires (generates), from the high speed picked up image, a standardluminance picked up image PL#i and a low luminance picked up image PH#iby the second acquisition method. Then, the saturated pixel restorationsection 33 performs a saturated pixel restoration process using thestandard luminance picked up image PL#i and the low luminance picked upimage PH#i.

In particular, referring to FIG. 18, the saturated pixel restorationsection 33 includes a saturation decision section 51, a restorationsection 52, a standard luminance picked up image generation section 61and a low luminance picked up image generation section 62.

Accordingly, the saturated pixel restoration section 33 of FIG. 18 iscommon to that in the case of FIG. 16 in that it includes the saturationdecision section 51 and the restoration section 52.

However, the saturated pixel restoration section 33 of FIG. 18 isdifferent from that in the case of FIG. 16 in that it includes thestandard luminance picked up image generation section 61 and the lowluminance picked up image generation section 62.

To the standard luminance picked up image generation section 61 and thelow luminance picked up image generation section 62, high speed pickedup images of the seven visual points obtained by high speed image pickupare supplied from the image pickup apparatus 11.

The standard luminance picked up image generation section 61 adds, foreach of the seven visual points, a predetermined number of high speedpicked up images from the image pickup apparatus 11 to generate astandard luminance picked up image PL#i and supplies the standardluminance picked up image PL#i to the saturation decision section 51 andthe restoration section 52.

The low luminance picked up image generation section 62 adds, for eachof the seven visual points, the number of high speed picked up imagesfrom the image pickup apparatus 11 smaller than the number of those bythe standard luminance picked up image generation section 61 to generatea low luminance picked up image PH#i and supplies the low luminancepicked up image PH#i to the saturation decision section 51 and therestoration section 52.

FIG. 19 is a flow chart illustrating an example of a saturated pixelrestoration process performed by the saturated pixel restoration section33 of FIG. 18.

At step S41, the standard luminance picked up image generation section61 adds, for each of the seven visual points, high speed picked upimages from the image pickup apparatus 11 to generate a standardluminance picked up image PL#i and supplies the standard luminancepicked up image PL#i to the saturation decision section 51 and therestoration section 52.

Further, at step S41, the low luminance picked up image generationsection 62 adds, for each of the seven visual points, high speed pickedup images from the image pickup apparatus 11 to generate a low luminancepicked up image PH#i and supplies the low luminance picked up image PH#ito the saturation decision section 51 and the restoration section 52.

Then, the processing advances from step S41 to step S42, and thereafter,processes similar to those at steps S31 to S38 in FIG. 17 are performedat steps S42 to S49, respectively.

FIG. 20 is a plan view depicting another configuration example of theimage pickup apparatus 11.

In particular, FIG. 20 depicts an example of a configuration of theimage pickup apparatus 11 where a standard luminance picked up imagePL#i and a low luminance picked up image PH#i in both of which the sameimage pickup object is reflected are acquired by the third acquisitionmethod.

In FIG. 20, the image pickup apparatus 11 is configured from 19 cameraunits.

In particular, in FIG. 20, one camera unit is set as a reference cameraunit, and five camera units are disposed in a horizontal directioncentering around the reference camera unit.

Further, above and below the five camera units centered at the referencecamera unit, individually four camera units juxtaposed in the horizontaldirection are disposed.

Further, above the four camera units at the upper side of the fivecamera units centered at the reference camera unit, three camera unitsjuxtaposed in the horizontal direction are disposed.

Furthermore, under the four camera units at the lower side of the fivecamera units centered at the reference camera unit, three camera unitsjuxtaposed in the horizontal direction are disposed.

Further, in FIG. 20, from among the 19 camera units configuring theimage pickup apparatus 11, four camera units neighboring at the leftupper side, left lower side, right upper side and right lower side ofthe reference camera unit are camera units with an ND (Neutral Density)filter on each of which an ND filter is mounted.

Here, in the following description, each camera unit on which no NDfilter is mounted is referred to as ordinary camera unit.

Referring to FIG. 20, reference symbol U1 denotes an ordinary cameraunit, and U2 denotes a camera unit with an ND filter.

While, according to the first and second acquisition methods, imagepickup is performed by a plural number of times to acquire a standardluminance picked up image PL#i and a low luminance picked up image PH#ithat reflect the same image pickup object but are different from eachother in exposure time period, according to the third acquisitionmethod, a standard luminance picked up image PL#i and a low luminancepicked up image PH#i are acquired by a single time of image pickup(one-shot image pickup).

In particular, in the third acquisition method, the 15=19−4 ordinarycamera units U1 of the image pickup apparatus 11 and the four cameraunits U2 with an ND filter perform image pickup, for example, for thestandard exposure time period.

By the ordinary camera units U1 performing image pickup for the standardexposure time period, a standard luminance picked up image for thestandard exposure time period can be acquired.

On the other hand, it is assumed now that the luminance of lightobserved through an ND filter by the four camera units U2 with an NDfilter is, for example, 1/16, 1/256, 1/4096 and 1/65536 that when no NDfilter is interposed. In other words, the sensitivity of the four cameraunits U2 with an ND filter is 1/16, 1/256, 1/4096 and 1/65536 that ofthe ordinary camera units U1.

In this case, when image pickup for the standard exposure time period isperformed by the four camera units U2 with an ND filter, a first lowluminance picked up image, a second low luminance picked up image, athird low luminance picked up image and a fourth low luminance picked upimage of exposure time periods that are equivalently 1/16, 1/256, 1/4096and 1/65536 the standard exposure time period, respectively, can beacquired.

Accordingly, the standard luminance picked up image and the first tofourth low luminance picked up images obtained by the third acquisitionmethod are images of different visual points picked up at the sametiming but with different sensitivities from each other.

FIG. 21 is a block diagram depicting a third configuration example ofthe saturated pixel restoration section 33.

In particular, FIG. 21 depicts an example of a configuration of thesaturated pixel restoration section 33 where a standard luminance pickedup image and a low luminance picked up image in both of which the sameimage pickup object is reflected are acquired by the third acquisitionmethod.

Referring to FIG. 21, the saturated pixel restoration section 33includes a parallax information acquisition section 71, a standardluminance picked up image generation section 72, a low luminance pickedup image generation section 73, a saturation decision section 74 and arestoration section 75.

The parallax information acquisition section 71 acquires parallax mapsof individual picked up images (standard luminance picked up image andfirst to fourth low luminance picked up images) picked up by the (19)camera units of the 19 visual points configuring the image pickupapparatus 11 (FIG. 20) and supplies the parallax maps to the standardluminance picked up image generation section 72 and the low luminancepicked up image generation section 73.

The parallax information acquisition section 71 can generate theparallax maps of the individual picked up images picked up by the cameraunits of the 19 visual points using the picked up images picked up bythe camera units of the 19 visual points in a similar manner as in thecase of the parallax information generation section 31 (FIG. 3).Alternatively, the parallax information acquisition section 71 canacquire parallax maps of the picked up images picked up by the cameraunits of the 19 visual points from the parallax information generationsection 31.

To the standard luminance picked up image generation section 72, theparallax maps are supplied from the parallax information acquisitionsection 71, and also standard luminance picked up images picked up bythe ordinary camera units U1 of the 15 visual points are supplied fromthe image pickup apparatus 11.

The standard luminance picked up image generation section 72 generatesstandard luminance picked up images of the four visual points of thefour camera units U2 with an ND filter (each hereinafter referred toalso as ND camera visual point) using standard luminance picked upimages of the 15 visual points of the 15 ordinary camera units U1 fromthe image pickup apparatus 11 (each hereinafter referred to also asordinary camera visual point) and the parallax maps from the parallaxinformation acquisition section 71 and supplies the generated standardluminance picked up images to the saturation decision section 74 and therestoration section 75 together with the standard luminance picked upimages of the 15 ordinary camera visual points from the image pickupapparatus 11.

In particular, the standard luminance picked up image generation section72 successively selects the pixels of the (first to fourth) lowluminance picked up images of the ND camera visual point as a noticedpixel and refers to the parallax map from the parallax informationacquisition section 71 to detect a corresponding pixel of the standardluminance picked up image of the ordinary camera visual pointcorresponding to the noticed pixel. Then, the standard luminance pickedup image generation section 72 adopts the pixel value of thecorresponding pixel of the standard luminance picked up image of theordinary camera visual point as a pixel value of the noticed pixel ofthe standard luminance picked up image of the ND camera visual point togenerate a standard luminance picked up image of the ND camera visualpoint.

It is to be noted that the corresponding pixel corresponding to thenoticed pixel of the low luminance picked up image of the ND cameravisual point can be detected from each of the standard luminance pickedup images of the 15 ordinary camera visual points.

As the pixel value of the noticed pixel of the ordinary luminance pickedup image of the ND camera visual point, the pixel value of thecorresponding pixel of the standard luminance picked up image, forexample, of the reference camera unit (camera unit at the center of the19 camera units) from among the 15 corresponding pixels detectedindividually from the standard luminance picked up images of the 15ordinary camera visual points can be adopted.

Further, as the pixel value of the noticed pixel of the ordinaryluminance picked up image of the ND camera visual point, an averagevalue of pixel values of corresponding pixels, for example, of a set inwhich the number of corresponding pixels is greatest from among sets ofcorresponding pixels having close pixel values to each other from amongthe 15 corresponding pixels detected from each of the standard luminancepicked up images of the 15 ordinary camera visual points can be adopted.

To the low luminance picked up image generation section 73, not only aparallax map is supplied from the parallax information acquisitionsection 71, but also the first to fourth low luminance picked up imagespicked up by the camera units U2 with an ND filter of the four ND cameravisual points are supplied from the image pickup apparatus 11.

The low luminance picked up image generation section 73 generates firstto fourth low luminance picked up images of the 15 ordinary cameravisual points using the first to fourth low luminance picked up imagesof the four ND camera visual points from the image pickup apparatus 11and a parallax map from the parallax information acquisition section 71and supplies the generated first to fourth low luminance picked upimages to the restoration section 75 together with the first to fourthlow luminance picked up images of the four ND camera visual points fromthe image pickup apparatus 11.

In particular, the low luminance picked up image generation section 73successively selects the pixels of the standard luminance picked upimage of each ordinary camera visual point as a noticed pixel and refersto the parallax map from the parallax information acquisition section 71to detect a corresponding pixel of each of the first to fourth lowluminance picked up images of the ND camera visual points correspondingto the noticed pixel. Then, the low luminance picked up image generationsection 73 adopts the pixel values of the corresponding pixels of thefirst to fourth low luminance picked up image of the ND camera visualpoints as the pixel values of the noticed pixels of the first to fourthlow luminance picked up images of the ordinary camera visual points togenerate first to fourth low luminance picked up images of the ordinarycamera visual points.

As described above, the standard luminance picked up image generationsection 72 generates standard luminance picked up images of the four NDcamera visual points and the low luminance picked up image generationsection 73 generates first to fourth low luminance picked up images ofthe 15 ordinary camera visual points. Consequently, standard luminancepicked up images and first to fourth low luminance picked up images canbe obtained from all of the 19 visual points of the 19 camera unitsconfiguring the image pickup apparatus 11 (FIG. 20).

It is to be noted that, while, in the present embodiment, standardluminance picked up images of the four ND camera visual points and firstto fourth low luminance picked up images of the 15 ordinary cameravisual points are generated in advance in order to facilitatedescription, the pixel values of standard luminance picked up images ofthe four ND camera visual points and the pixel values of the first tofourth low luminance picked up images of the 15 ordinary camera visualpoints can be generated when necessary for a pixel or pixels whichrequire such pixel values.

The saturation decision section 74 performs a saturation decision fordeciding whether or not each pixel of the standard luminance picked upimages of the 19 visual points from the standard luminance picked upimage generation section 72 is a saturated pixel and supplies a decisionresult (saturation decision result) of the saturation decision to therestoration section 75.

To the restoration section 75, a decision result of a saturationdecision is supplied from the saturation decision section 74 asdescribed hereinabove. Further, to the restoration section 75, standardluminance picked up images of the 19 visual points are supplied from thestandard luminance picked up image generation section 72 and first tofourth low luminance picked up images of the 19 visual points aresupplied from the low luminance picked up image generation section 73 asdescribed hereinabove.

Further, to the restoration section 75, restoration gains for first tofourth low luminance picked up images are supplied, for example, fromthe image pickup apparatus 11 of FIG. 20.

Here, the restoration gain for the first low luminance picked up imageis a restoration gain described hereinabove with reference to FIG. 17,which is used to perform restoration of a saturated pixel using thefirst low luminance picked up image.

Accordingly, the restoration gain for the first low luminance picked upimage is a value of the ratio between the exposure time period (standardexposure time period) of the standard luminance picked up image and theexposure time period of the first low luminance picked up image.

For example, if the exposure time period of the first low luminancepicked up image is (equivalently) 1/16 time the standard exposure timeperiod as described hereinabove with reference to FIG. 20, then therestoration gain for the first low luminance picked up image is16=1/(1/16) times.

Also the restoration gains for the second to fourth low luminance pickedup images can be determined similarly to the restoration gain for thefirst low luminance picked up image.

The restoration section 75 specifies, for each of the standard luminancepicked up images of the 19 visual points from the standard luminancepicked up image generation section 72, a saturated pixel in response toa saturation decision result from the saturation decision section 74.

Further, the restoration section 75 restores pixel values of saturatedpixels using restoration gains for the first to fourth low luminancepicked up images from the low luminance picked up image generationsection 73 and the first to fourth low luminance picked up images fromthe image pickup apparatus 11 (FIG. 20) as occasion demands and suppliesHDR picked up images of a higher dynamic range than that of the standardluminance picked up images to the incident ray reproduction section 36(FIG. 3), the HDR picked up images being obtained by restoration.

FIG. 22 is a view illustrating an example of correction of a parallax ofa parallax map.

In the saturated pixel restoration section 33 of FIG. 21, the standardluminance picked up image generation section 72 refers to a parallax mapacquired by the parallax information acquisition section 71 to detect acorresponding pixel of a standard luminance picked up image of theordinary camera visual point corresponding to a noticed pixel of a lowluminance picked up image of the ND camera visual point. Further, thelow luminance picked up image generation section 73 refers to theparallax map acquired by the parallax information acquisition section 71to detect corresponding pixels of (first to fourth) low luminance pickedup images of the ND camera visual points corresponding to the noticedpixel of the standard luminance picked up image of the ordinary cameravisual point.

A parallax registered in a parallax map that is referred to when acorresponding pixel of an image of one visual point corresponding to anoticed pixel of the other one visual point is to be detected asdescribed above by the standard luminance picked up image generationsection 72 and the low luminance picked up image generation section 73is corrected as occasion demands.

Here, it is assumed that the image pickup apparatus 11 is configuredfrom three camera units Ua, Ub and Uc juxtaposed in the horizontaldirection as depicted in FIG. 22 in order to simplify the description.

Further, it is assumed that, for example, the camera unit Ua at the mostleft side from among the three camera units Ua, Ub and Uc is determinedas a reference camera unit that picks up a reference image of a targetfor which a multilayer parallax map is to be generated.

Further, it is assumed that the reference camera unit Ua and the cameraunit Uc at the most right side are ordinary camera units and the centralcamera unit Ub is a camera unit with an ND filter. The reference cameraunit Ua and the ordinary camera unit Uc pick up a standard luminancepicked up image and the camera unit Ub with an ND filter picks up a lowluminance picked up image.

Here, the standard luminance picked up image picked up by the referencecamera unit Ua is referred to as reference image Ua or standardluminance picked up image Ua. Meanwhile, a low luminance picked up imagepicked up by the camera unit Ub with an ND filter is referred to also aslow luminance picked up image Ub. Further, the standard luminance pickedup image picked up by the ordinary camera unit Uc is referred to also asstandard luminance picked up image Uc.

Now, it is assumed that the parallax information acquisition section 71generates a parallax map (reference parallax map) of the standardluminance picked up image (reference image) Ua picked up by thereference camera unit Ua using a baseline length (distance betweenoptical axes) between the reference camera unit Ua and the ordinarycamera unit Uc as a reference baseline length.

The parallax map of a visual point other than the visual point(reference visual point) of the reference camera unit Ua, namely, theparallax map of the standard luminance picked up image Uc picked up, forexample, by the ordinary camera unit Uc can be generated simply andeasily utilizing a reference parallax map as described hereinabove withreference to FIGS. 7 and 8.

In particular, the parallax map of the standard luminance picked upimage Uc picked up by the ordinary camera unit Uc can be generated bymoving, on the reference parallax map, the parallax registered at theposition of each pixel in the camera position relation directionaccording to the positional relationship between the reference cameraunit Ua and the ordinary camera unit Uc by a distance equal to theparallax as described hereinabove with reference to FIG. 7, and theninterpolating an interpolation parallax in a non-registration area asdescribed hereinabove with reference to FIG. 8.

The parallax generated utilizing the reference parallax map andregistered in the parallax map of the standard luminance picked up imageUc is a parallax regarding two points spaced by the reference baselinelength that is a baseline length between the reference camera unit Uaand the ordinary camera unit Uc.

Therefore, when a corresponding pixel of the low luminance picked upimage Ub corresponding to a noticed pixel of the standard luminancepicked up image Uc is to be detected, the parallax registered in theparallax map of the standard luminance picked up image Uc is correctedso as to become a parallax regarding two points spaced from each otherby a baseline length between the ordinary camera unit Uc that picks upthe standard luminance picked up image Uc and the camera unit Ub with anND filter that picks up the low luminance picked up image Ub.

In particular, if it is assumed now that the reference baseline lengththat is the baseline length between the reference camera unit Ua and theordinary camera unit Uc is represented by ac and the baseline lengthbetween camera unit Ub with an ND filter and the ordinary camera unit Ucis represented by bc, then the parallax registered in the parallax mapof the standard luminance picked up image Uc is corrected so as to bemultiplied by the value bc/ac of the ratio between the baseline lengthbc and the reference baseline length.

For example, it is assumed that both the baseline length ab between thereference camera unit Ua and the camera unit Ub with an ND filter andthe baseline length bc between the camera unit Ub with an ND filter andthe ordinary camera unit Uc are 5 mm.

In this case, the reference baseline length that is the baseline lengthac between the reference camera unit Ua and the ordinary camera unit Ucis 10 mm.

Now, if it is assumed that the parallax of the noticed pixel generatedutilizing the reference parallax map and registered in the parallax mapof the standard luminance picked up image Uc is, for example, 10(pixels), then the parallax of 10 is multiplied by the value 5/10 of theratio between the baseline length bc=5 mm and the reference baselinelength ac=10 mm and corrected to the parallax of 5.

Then, as the corresponding pixel of the low luminance picked up image Ubcorresponding to the noticed pixel of the standard luminance picked upimage Uc, a pixel of the low luminance picked up image Ub at a positiondisplaced by the parallax of 5 from the position of the noticed pixel isdetected.

FIG. 23 is a flow chart illustrating an example of the saturated pixelrestoration process performed by the saturated pixel restoration section33 of FIG. 21.

At step S61, the parallax information acquisition section 71 acquiresparallax maps of picked up images (standard luminance picked up imageand first to fourth low luminance picked up images) picked up by thecamera units of the 19 visual points configuring the image pickupapparatus 11 (FIG. 20) and supplies the parallax maps to the standardluminance picked up image generation section 72 and the low luminancepicked up image generation section 73.

Further, at step S61, the standard luminance picked up image generationsection 72 generates standard luminance picked up images of the four NDcamera visual points using the standard luminance picked up images ofthe 15 ordinary camera visual points from among the camera units of the19 visual points configuring the image pickup apparatus 11 (FIG. 20) andthe parallax maps from the parallax information acquisition section 71and supplies the generated standard luminance picked up images to thesaturation decision section 74 and the restoration section 75 togetherwith the standard luminance picked up images of the 15 ordinary cameravisual points.

Furthermore, at step S61, the low luminance picked up image generationsection 73 generates first to fourth low luminance picked up images ofthe 15 ordinary camera visual points using the first to fourth lowluminance picked up images of the four ND camera visual points from theimage pickup apparatus 11 and the parallax maps from the parallaxinformation acquisition section 71 and supplies the generated first tofourth low luminance picked up images to the restoration section 75together with the first to fourth low luminance picked up images of thefour ND camera visual points.

Then, the processing advances from step S61 to step S62, at which thesaturation decision section 74 selects, from among the 19 visual pointsof the 19 camera units configuring the image pickup apparatus 11 (FIG.20), one visual point that has not been selected as a noticed visualpoint as yet as a noticed visual point. Thereafter, the processingadvances to step S63.

At step S63, the saturation decision section 74 selects, from among thepixels of the standard luminance picked up image of the noticed visualpoint from among the standard luminance picked up images of the 19visual points supplied from the standard luminance picked up imagegeneration section 72, one pixel that has not been selected as a noticedpixel as yet as a noticed pixel. Thereafter, the processing advances tostep S64.

At step S64, a process for acquiring the pixel value of the noticedpixel of the HDR picked up image of the noticed visual point (pixel atthe same position as that of the noticed pixel of the standard luminancepicked up image of the noticed visual point) is performed. Then, theprocessing advances to step S65.

At step S65, the saturation decision section 74 decides whether or notall of the pixels of the standard luminance picked up image of thenoticed visual point have been selected as a noticed pixel.

If it is decided at step S65 that all of the pixels of the standardluminance picked up image of the noticed visual point have not beenselected as a noticed pixel, then the processing returns to step S63,and thereafter, similar processes are repeated.

On the other hand, if it is decided at step S65 that all of the pixelsof the standard luminance picked up image of the noticed visual pointhave been selected as a noticed pixel, then the processing advances tostep S66.

At step S66, the saturation decision section 74 decides whether or notall of the 19 visual points of the 19 camera units configuring the imagepickup apparatus 11 (FIG. 20) have been selected as a noticed visualpoint.

If it is decided at step S66 that all of the 19 visual points have notbeen selected as a noticed visual point as yet, then the processingreturns to step S62, and thereafter, similar processes are repeated.

On the other hand, if it is decided at step S66 that all of the 19visual points have been selected as a noticed visual point, then therestoration section 52 supplies the HDR picked up images of the 19visual points obtained by the processes described above to the incidentray reproduction section 36 (FIG. 3), thereby ending the saturated pixelrestoration process.

It is to be noted that, while, in FIG. 23, the saturated pixelrestoration process is performed for all of the 19 visual points of the19 camera units configuring the image pickup apparatus 11, the saturatedpixel restoration process can be performed only for the 15 ordinarycamera visual points from among the 19 visual points.

In this case, the HDR picked up images obtained by the saturated pixelrestoration process are not the HDR picked up images of the 19 visualpoints but are the HDR picked up images of the 15 ordinary camera visualpoints, the saturated pixel restoration section 33 of FIG. 21 can beconfigured without provision of the standard luminance picked up imagegeneration section 72.

FIG. 24 is a flow chart illustrating an example of a process foracquiring a pixel value of a noticed pixel of an HDR picked up image ofa noticed visual point performed at step S64 of FIG. 23.

At step S71, the saturation decision section 74 acquires a pixel valueof a noticed pixel of a standard luminance picked up image of a noticedvisual point from the standard luminance picked up image generationsection 72 as a standard pixel value of the noticed pixel. Then, theprocessing advances to step S72.

At step S72, the saturation decision section 74 decides a saturationdecision of whether or not the standard pixel value of the noticed pixelof the standard luminance picked up image of the noticed visual point isequal to or higher than a threshold value TH1.

If it is decided at step S72 that the standard pixel value of thenoticed pixel of the standard luminance picked up image of the noticedvisual point is not equal to or higher than the threshold value TH1,namely, if the standard pixel value of the noticed pixel of the standardluminance picked up image of the noticed visual point is not in asaturated state, then the saturation decision section 74 supplies asaturation decision result that the standard pixel value is not in asaturated state to the restoration section 75 (FIG. 21). Then, theprocessing advances to step S73.

At step S73, the restoration section 75 selects, in response to thesaturation decision result from the saturation decision section 74 thatthe standard pixel value is not in a saturated state, the standard pixelvalue of the noticed pixel of the standard luminance picked up image ofthe noticed visual point from the standard luminance picked up imagegeneration section 72 as a pixel value of the noticed pixel of the HDRpicked up image of the noticed visual point (pixel at a position same asthat of the noticed pixel of the standard luminance picked up image ofthe noticed visual point). Thereafter, the processing returns.

On the other hand, if it is decided at step S72 that the standard pixelvalue of the noticed pixel of the standard luminance picked up image ofthe noticed visual point is equal to or higher than the threshold valueTH1, namely, if the standard pixel value of the noticed pixel of thestandard luminance picked up image of the noticed visual point is in asaturated state or may be in a saturated state with a high degree ofpossibility, the saturation decision section 74 supplies a saturationdecision result that the standard pixel value is in a saturated state tothe restoration section 75. Then, the processing advances to step S74.

At step S74, the restoration section 75 acquires, in response to thesaturation decision result that the standard pixel value is in asaturated state from the saturation decision section 74, pixel values ofnoticed pixels of the respective first to fourth low luminance picked upimages of the noticed visual point (pixels at the same position as thatof the noticed pixel of the standard luminance picked up image of thenoticed visual point) from the low luminance picked up image generationsection 73 as first to fourth low luminance pixel values v1 to v4 of thenoticed pixel.

Further, the restoration section 75 acquires restoration gains g1 to g4for the first to fourth low luminance picked up images from the imagepickup apparatus 11. Then, the processing advances from step S74 to stepS75.

At step S75, the restoration section 75 decides whether or not the firstlow luminance pixel value v1 that is the pixel value of the noticedpixel of the first low luminance picked up image of the noticed visualpoint satisfies an expression TH2<v1<TH1 that uses a threshold value TH1and another threshold value TH2 that is lower than the threshold valueTH1.

If it is decided at step S75 that the first low luminance pixel value v1satisfies the expression TH2<v1<TH1, namely, if the first low luminancepixel value v1 is not such a low value as that of noise and besides isnot in a saturated state, then the processing advances to step S76.

At step S76, the restoration section 75 multiplies the first lowluminance pixel value v1 that is the pixel value of the noticed pixel ofthe first low luminance picked up image of the noticed visual point bythe restoration gain g1 for the first low luminance picked up image anddetermines a resulting pixel value of the multiplication as arestoration pixel value when the saturated pixel is restored. Further,the restoration section 75 selects the restoration pixel value as apixel value of the noticed pixel of the HDR picked up image of thenoticed visual point. Then, the processing returns.

On the other hand, if it is decided at step S75 that the first lowluminance pixel value v1 does not satisfy the expression TH2<v1<TH1,then the processing advances to step S77.

At step S77, the restoration section 75 decides whether or not thesecond low luminance pixel value v2 that is the pixel value of thenoticed pixel of the second low luminance picked up image of the noticedvisual point satisfies the expression TH2<v2<TH1.

If it is decided at step S77 that the second low luminance pixel valuev2 satisfies the expression TH2<v2<TH1, namely, if the second lowluminance pixel value v2 is not such a low value as that of noise andbesides is not in a saturated state, then the processing advances tostep S78.

At step S78, the restoration section 75 multiplies the second lowluminance pixel value v2 that is the pixel value of the noticed pixel ofthe second low luminance picked up image of the noticed visual point bythe restoration gain g2 for the second low luminance picked up image anddetermines the resulting pixel value as a restoration pixel valuerestored from that of the saturated pixel. Further, the restorationsection 75 selects the restoration pixel value as the pixel value of thenoticed pixel of the HDR picked up image of the noticed visual point.Thereafter, the processing is returned.

On the other hand, if it is decided at step S77 that the second lowluminance pixel value v2 does not satisfy the expression TH2<v2<TH1,then the processing advances to step S79.

At step S79, the restoration section 75 decides whether or not the thirdlow luminance pixel value v3 that is the pixel value of the noticedpixel of the third low luminance picked up image of the noticed visualpoint satisfies an expression TH2<v3<TH1.

If it is decided at step S79 that the third low luminance pixel value v3satisfies the expression TH2<v3<TH1, namely, if the third low luminancepixel value v3 is not such a low value as that of noise and besides isnot in a saturated state, then the processing advances to step S80.

At step S80, the restoration section 75 multiplies the third lowluminance pixel value v3 that is the pixel value of the noticed pixel ofthe third low luminance picked up image of the noticed visual point bythe restoration gain g3 for the third low luminance picked up image anddetermines the resulting pixel value as the restoration pixel valuerestored from that of the saturated pixel. Further, the restorationsection 75 selects the restoration pixel value as the pixel value of thenoticed pixel of the HDR picked up image of the noticed visual point.Thereafter, the processing returns.

On the other hand, if it decided at step S79 that the third lowluminance pixel value v3 does not satisfy the expression TH2<v3<TH1,then the processing advances to step S81.

At step S81, the restoration section 75 multiplies the fourth lowluminance pixel value v4 that is the pixel value of the noticed pixel ofthe fourth low luminance picked up image of the noticed visual point bythe restoration gain g4 for the fourth low luminance picked up image anddetermines the resulting pixel value as the restoration pixel valuerestored from that of the saturated pixel. Further, the restorationsection 75 selects the restoration pixel value as the pixel value of thenoticed pixel of the HDR picked up image of the noticed visual point.Thereafter, the processing returns.

It is to be noted that the (HDR) picked up images of the plurality ofvisual points of the high dynamic range obtained as a result of thesaturated pixel restoration process can be made a target not only forgeneration of parallax information by the parallax informationgeneration section 31 and generation of an emulation image by the lensemulation section 35 but also for an arbitrary image process for whichpicked up images of a plurality of visual points are required.

Further, if the saturated pixel restoration process is performed notonly for picked up images of a plurality of visual points picked up bythe image pickup apparatus 11 having the plurality of camera units 21_(i) or the like but also for arbitrary images to which the light fieldtechnology can be applied, then a clear blur can be reproduced.

As a method for picking up an image to which the light field technologycan be applied, in addition to a method for picking up picked up imagesof a plurality of visual points using the image pickup apparatus 11having the plurality of camera units 21 _(i), for example, a method forperforming image pickup using an MLA (Micro Lens Array) disclosed, forexample, in Ren. Ng. and seven others, “Light Field Photography with aHand-Held Plenoptic Camera,” Stanford Tech Report CTSR 2005-02 isavailable.

<Outline of Lens Emulation Process of Lens Emulation Section 35>

FIG. 25 is a view illustrating an outline of a lens emulation process ofthe lens emulation section 35 of FIG. 3.

In the lens emulation process, the incident ray reproduction section 36(FIG. 3) reproduces rays incident to a virtual lens from among raysemitted from a real space point such as a point on an object existing ina real space which has become an image pickup object upon image pickupby the image pickup apparatus 11 (including not only light emitted fromthe real space point where the real space point emits light but alsoreflected light reflected by the real space point).

The virtual lens is a virtual lens having a synthetic aperture providedby the camera units 21 ₁ to 21 ₇ configuring the image pickup apparatus11 (FIG. 2), and the entity of the virtual lens is the camera units 21 ₁to 21 ₇.

Further, by the lens emulation process, lens information (emulation lensinformation) that defines rays that pass the emulation lens is generatedby the emulation lens information generation section 37 (FIG. 3).

As described hereinabove with reference to FIG. 3, the emulation lensmay be an optical lens that actually exists or may be an optical lensthat does not exist actually.

Further, the lens information includes a PSF intensity distributionrepresentative of a response of the emulation lens to a point lightsource and so forth.

In the lens emulation process, the light condensing processing section38 (refer to FIG. 3) performs a digital signal process as a lightcondensing process for condensing rays reproduced by the incident rayreproduction section 36 using lens information obtained by the emulationlens information generation section 37 on the virtual sensor through theemulation lens.

The entity of the virtual sensor is, for example, a memory not depicted,and in the light condensing process, lens information is used to add avalue corresponding to the luminance of rays to (a storage value of) thememory to generate an emulation image.

FIG. 26 is a view illustrating a light condensing process by an actualoptical lens and a light condensing process of the lens emulationprocess.

A of FIG. 26 depicts a light condensing process by an actual opticallens.

The actual optical lens samples a large number of rays emitted from anobject in a real space to form an image on an image plane in accordancewith lens characteristics of the actual optical lens.

In the actual optical lens, the angle of rays to be sampled by theoptical lens varies, for example, depending upon the aperture.

In particular, if the aperture is restricted, then rays that are spreadat a large angle w with respect to the optical axis from the object arenot sampled by the optical lens. On the other hand, if the aperture isopened, then rays spread at a large angle w with respect to the opticalaxis from the object are sampled by the optical lens.

An image picA of A of FIG. 26 is an image picked up with the aperturerestricted and is an image in which the depth of field is deep and whichis generally in focus. Further, in the image picA, although a bulbexists behind a character of a child in a right upper region, raysspread at a large angle with respect to the optical axis from the bulbare not sampled by the optical lens, and therefore, the bulb is notreflected behind the character of the child.

Another image picB of A of FIG. 26 is an image picked up with theaperture opened and is an image in which the depth of field is shallowand which is in focus only at a portion thereof and is blurred at theother most part thereof. Further, in the image picB, the bulb existsbehind the character of the child in the right upper region, and raysspread at a large angle with respect to the optical axis from the bulbare sampled by the optical lens. Therefore, part of the bulb isreflected behind the character of the child.

B of FIG. 26 depicts a light condensing process of the lens emulationprocess.

In the light condensing process of the lens emulation process, raysemitted from an object in a real space and imaged (recorded) by theplurality of camera units 21 _(i) of the image pickup apparatus 11 areused to reproduce (generate) rays that are incident to the virtual lenshaving a synthetic aperture provided by the plurality of camera units 21_(i).

Here, in B of FIG. 26, three rays are imaged by the three camera units21 ₁, 21 ₂ and 21 ₅ as a plurality of camera units. Further, rays to beincident to the virtual lens are reproduced such that rays among thethree rays are interpolated.

In the light condensing process of the lens emulation process, afterrays to be incident to the virtual lens are reproduced in such a manneras described above, the rays are condensed on the virtual sensor inaccordance with the lens information of the emulation lens.Consequently, in the emulation image obtained as a result of the lightcondensing, a blur degree similar to that where an image is picked upactually using the emulation lens is reproduced.

<Reproduction of Rays Incident to Virtual Lens>

FIG. 27 is a block diagram depicting an example of a configuration ofthe incident ray reproduction section 36 of FIG. 3.

Referring to FIG. 27, the incident ray reproduction section 36 includesa real space point selection section 101, a ray generation section 102,a collision decision section 103 and a luminance allocation section 104.

To the real space point selection section 101, parallax maps aresupplied from the parallax information generation section 31.

The real space point selection section 101 selects a space point in areal space whose image is picked up by the image pickup apparatus 11using a multilayer parallax map from among the parallax maps from theparallax information generation section 31 as a noticed real space pointand supplies the noticed real space point to the ray generation section102.

The ray generation section 102 generates (straight lines as) rays to beincident to the virtual lens from the noticed real space point from thereal space point selection section 101 and supplies the rays to thecollision decision section 103.

To the collision decision section 103, not only rays are supplied fromthe collision decision section 103, but also parallax maps are suppliedfrom the parallax information generation section 31.

The collision decision section 103 uses the multilayer parallax map fromamong the parallax maps from the parallax information generation section31 to perform a collision decision for deciding whether or not the raysfrom the collision decision section 103 collide with an object in thereal space before they enter the virtual lens.

Then, the collision decision section 103 supplies rays that remain as aresult of the collision decision to the luminance allocation section104.

To the luminance allocation section 104, not only the rays are suppliedfrom the collision decision section 103, but also parallax maps aresupplied from the parallax information generation section 31, and also(HDR) picked up images HD#i of the seven visual points as plural visualpoints are supplied from the saturated pixel restoration section 33.

The luminance allocation section 104 uses the parallax maps from theparallax information generation section 31 and the picked up images HD#ifrom the saturated pixel restoration section 33 to allocate a luminanceto the rays from the collision decision section 103, namely, to the raysremaining as a result of the collision decision, and supplies the raysafter the allocation of the luminance to the light condensing processingsection 38 (FIG. 3).

FIG. 28 is a view illustrating a real space point.

In particular, FIG. 28 is a schematic plan view when a real space whoseimage is picked up by a camera unit 21 _(i) configuring the image pickupapparatus 11 serving as the virtual lens is viewed from above.

Here, as a three-dimensional coordinate system that defines a positionin the real space (real space point), a three-dimensional coordinatesystem is used which has the origin at the principal point of thevirtual lens or the emulation lens and has an x axis and a y axis alonga horizontal direction and a vertical direction, respectively, when theimage pickup apparatus 11 (FIG. 2) is viewed from the front and a z axisalong a depthwise direction from the origin (direction of an imagepickup object).

A real space point (x, y, z) that is a position in the real space of anobject (image pickup object) reflected at a certain pixel p of areference image can be determined from the position of the pixel p onthe reference image (position on an image sensor not depicted of thecamera unit 21 ₁) and the parallax d of the pixel p.

Therefore, the real space point selection section 101 determines a realspace point corresponding to the pixel p having the parallax d (positionin the real space of an object that may be reflected at the pixel p)from the position and the parallax d of the pixel p registered in themultilayer parallax map.

Now, if it is assumed that (a set of) real space points having theparallax d registered in the multilayer parallax map are referred to asparallax registration position, then the real space point selectionsection 101 successively selects the real space points configuring theparallax registration position as a noticed real space point.

It is to be noted that, in FIG. 28, THETA denotes an angle of view inthe horizontal direction of the reference image (reference camera unit21 ₁). The parallax registration position exists within a range that isspread by the angle THETA of view centered at the optical axis of thevirtual lens.

The axis of the virtual lens is a straight line that passes the centerof the reference image and is perpendicular to the reference image(optical axis of the reference camera unit 21 ₁).

FIG. 29 is a view illustrating a determination method for determining areal space point using a multilayer parallax map.

It is assumed now that, as a multilayer parallax map space forrepresenting a multilayer parallax map, a three-dimensional space isused which has an x axis and a y axis at the positions of the referenceimage in the horizontal direction and the vertical direction and has a zaxis by which a value that can be assumed by a parallax obtained by theparallax information generation section 31 (FIG. 3) is represented.

In such a multilayer parallax map space as described above, a parallax dof a pixel at the position (x, y) can be registered by making a parallaxflag representing that a parallax is registered at the position (x, y,d).

Here, in the present embodiment, a maximum value of the parallax thatcan be registered into the multilayer parallax map is represented asDmax and a minimum value is represented as Dmin. In this case, the sizeof the multilayer parallax map space in the z-axis direction isDmax−Dmin+1. It is to be noted that, as Dmin, for example, 0 (infinity)can be adopted.

Further, each parallax d registered in the multilayer parallax map canbe converted into a distance z in the real space in the depthwisedirection from the principal point of the virtual lens (reference cameraunit 21 ₁), for example, in accordance with an expression z=37.4/d.

It is to be noted that the expression for converting the parallax d intothe distance z is not limited to the expression z=37.4/d and differsdepending upon the resolution, angle of view and focal length of thereference camera unit 21 ₁.

Now, if the pixel p of the reference image whose x coordinate is Xpic isnoticed as noticed pixel p, then in the multilayer parallax map of FIG.29, parallaxes D₁ and D₂ are registered for the noticed pixel p.

The real space point selection section 101 successively selects theparallaxes D₁ and D₂ for the noticed pixel p as a noticed parallax to benoticed and selects a real space point corresponding to the noticedpixel p having the noticed parallax as noticed real space point.

Now, it is assumed that, from between the parallaxes D₁ and D₂, theparallax D₁ is selected as noticed parallax.

Further, the number of pixels in the horizontal direction (x-axisdirection) of the reference image (parallax map) is represented as widthand the angle of view in the horizontal direction of the reference imageis represented by THEATA. Furthermore, the position (distance) of a realspace point P1, which corresponds to the noticed pixel p having thereference parallax D₁, in the x-axis direction from the optical axis isrepresented as x1.

The real space point selection section 101 first converts the referenceparallax D₁ into the distance z=z1 in the real space.

Then, the real space point selection section 101 uses the distance z=z1corresponding to the reference parallax D₁ to determine a position(distance) x1 of the real space point P1, which corresponds to thenoticed pixel p having the reference parallax D₁, in the x-axisdirection from the optical axis.

In particular, the distance x1 in the real space and the number ofpixels Xpic-width/2 in the multilayer parallax map space correspond toeach other. Further, the distance z1×tan(THEATA/2) representing one halfthe angle of view in the horizontal direction in the real space and thenumber of pixels width/2 representative of one half the angle of view inthe horizontal direction in the multilayer parallax map space correspondto each other.

Since the ratio between x1 and Xpic−width/2 and the ratio betweenz1×tan(THEATA/2) and width/2 coincide with each other, an expression x1:Xpic−width/2=z1×tan(THEATA/2): width/2 is satisfied.

Accordingly, the position x1 of the real space point P1, whichcorresponds to the noticed pixel p having the reference parallax D₁, inthe x-axis direction from the optical axis can be determined inaccordance with the expressionx1=((Xpix−width/2)(z1×tan(THEATA/2))/(width/2).

The real space point selection section 101 determines the position x1 ofthe real space point P1, which corresponds to the noticed pixel p havingthe reference parallax D₁, in the x-axis direction from the optical axisin such a manner as described above.

The real space point selection section 101 similarly determines theposition of the real space point P1, which corresponds to the noticedpixel p having the reference parallax D₁, in the y-axis direction fromthe optical axis thereby to determine (the xyz coordinates of) the realspace point P1 corresponding to the noticed pixel p having the referenceparallax D₁.

Also the real space point corresponding to the pixel p having theparallax D₂ can be determined in a similar manner.

FIG. 30 is a view illustrating an example of generation rays performedby the ray generation section 102 of FIG. 27.

In particular, FIG. 30 is a front elevational view of the virtual lensas viewed from the front (image pickup object side).

The ray generation section 102 sets an area including (the syntheticaperture of) the virtual lens as lens area.

In FIG. 30, for example, a minimum rectangular area surrounding thevirtual lens is set as a lens area.

The ray generation section 102 divides (the virtual lens surrounded by)the lens area into lens area units that are small regions and performs,considering a real space point as a point light source, generation of aray to be incident to (for example, the center of) each lens area unitfrom the real space point as point light source, namely, calculation ofa straight line as a ray to be incident to each lens area unit from thereal space point.

In FIG. 30, the lens area is divided into totaling Lx×Ly lens area unitsincluding Lx lens area units in the horizontal direction and Ly lensarea units in the vertical direction.

In this case, the ray generation section 102 generates, in regard to onereal space point, Lx×Ly straight lines individually interconnecting thereal space point and the Lx×Ly lens area units densely as rays to beincident to the virtual lens.

Here, it is assumed that the distance between (the centers of) the lensarea units positioned adjacent each other in the horizontal direction orthe vertical direction is referred to as angle resolution with which anangle between two rays emitted from the real space point can bedistinguished.

For example, if it is assumed that the synthetic aperture (diameter ofthe virtual lens) is 40 mm and the numbers Lx and Ly of the lens areaunits in the horizontal direction and the vertical direction of the lensarea are 21, then the angle resolution is 40/21 mm.

Further, a grid point that is a cross point between a straight line inthe horizontal direction and another straight line in the verticaldirection by which the lens area is divided into the lens area units isreferred to also as grid point LP#i (i=1, 2, . . . , (Lx+1) (Ly+1)).

The distance between grid points LP#i and LP#j positioned adjacent eachother in the horizontal direction or the vertical direction representsthe angle resolution.

FIG. 31 is a view illustrating a collision decision performed by thecollision decision section 103 of FIG. 27 and allocation of a luminanceto a ray performed by the luminance allocation section 104.

In particular, FIG. 31 is a schematic plan view when a real space whoseimage is picked up by the camera units 21 _(i) configuring the imagepickup apparatus 11 as the virtual lens is viewed from above.

The collision decision section 103 performs a collision decision fordeciding whether or not the Lx×Ly rays from the collision decisionsection 103, which are emitted from the real space point and directedtoward the Lx×Ly lens area units of the virtual lens, collide with anobject in the real space before they enter the virtual lens using themultilayer parallax map.

In particular, if a ray emitted from the real space point and directedtoward a lens area unit of the virtual lens collides (crosses) with aparallax registration position before it enters the lens area unit, thenthe collision decision section 103 decides that the ray collides.

On the other hand, if a ray emitted from the real space point anddirected toward a lens area unit of the virtual lens does not collidewith a parallax registration position before it enters the lens areaunit, then the collision decision section 103 decides that the ray doesnot collide.

Then, the collision decision section 103 supplies those rays that remainas a result of the collision decision, namely, the rays that aredetermined not to collide, to the luminance allocation section 104.

It is to be noted that the collision decision section 103 allocates theparallax d=D corresponding to the real space point (x, y, z) from whichthe ray is emitted to each of the rays that remain as a result of thecollision decision and allocates, to each ray decided to collide, theparallax d=D′ corresponding to the parallax registration position atwhich the ray collides.

Whether a ray emitted from a certain real space point (x, y, z) entersthe virtual lens without colliding with an object after the collisiondecision can be recognized depending upon whether the parallax allocatedto the ray coincides with the parallax of the real space point (x, y, z)from which the ray is emitted.

In particular, when the parallax allocated to the ray coincides with theparallax of the real space point (x, y, z) from which the ray isemitted, the ray does not collide with an object and enters the virtuallens. On the other hand, if the parallax allocated to the ray does notcoincide with the parallax of the real space point (x, y, z) from whichthe ray is emitted, then the ray collides with an object at thedepthwise position corresponding to the parallax allocated to the rayand does not reach the virtual lens.

The luminance allocation section 104 allocates a luminance to raysremaining as a result of the collision decision from the collisiondecision section 103 using the multilayer parallax map and the picked upimages HD#i.

In particular, the luminance allocation section 104 determines acorresponding pixel corresponding to the real space point (x, y, z) fromwhich the rays remaining as a result of the collision decision areemitted for each of the picked up images HD1 to HD7 of the seven visualpoints.

Further, the luminance allocation section 104 refers to the parallaxmaps to detect, from among the corresponding pixels of the picked upimages HD1 to HD7, each pixel with regard to which a parallax coincidentwith the parallax d=D corresponding to the depth z of the real spacepoint (x, y, z) is registered as a ray luminance allocation pixel to beused for allocation of a luminance.

Then, the luminance allocation section 104 allocates a luminance to theray using values of R (Red), G (Green) and B (Blue) as a pixel value ofthe ray luminance allocation pixel.

In particular, the luminance allocation section 104 allocates, forexample, an average value of the pixel values (values of R, G and B) ofthe ray luminance allocation pixel as a luminance of the ray to the ray.

As described above, when a ray emitted from the real space point (x, y,z) collides with an object and does not enter the virtual lens, thecollision decision section 103 allocates a parallax, which correspondsto a parallax registration position at which the ray collides, to theray.

On the other hand, if a ray emitted from the real space point (x, y, z)does not collide with an object and enters the virtual lens, then thecollision decision section 103 allocates a parallax corresponding to thereal space point (x, y, z) from which the ray is emitted to the ray.

Further, to a ray that does not collide with an object and enters thevirtual lens, the luminance allocation section 104 allocates a pixelvalue (values of R, G and B) as a luminance.

FIG. 32 is a view schematically depicting a maximum number of dataobtained by the incident ray reproduction process performed by theincident ray reproduction section 36 of FIG. 27.

Now, it is assumed that the reference image HD1 is configured from Npixels pix1, pix2, . . . , pix#N and the number of parallaxes d that canbe registered into a parallax map (multilayer parallax map) isDPN=Dmax−Dmin+1 integral values in 1 pixel increments from the minimumvalue Dmin to the maximum value Dmax.

In this case, in the incident ray reproduction process, a pixel valuetable is registered for a real space point corresponding to acombination (pix#n, d) of an arbitrary pixel pix#n from among the Npixels pix1, pix2, . . . , pix#N and an arbitrary parallax d from amongthe DPN parallaxes Dmin, Dmin+1, . . . , Dmax at most as depicted inFIG. 32.

Into the pixel value table for the real space point corresponding to thecombination (pix#n, d), a parallax D allocated to a ray heading from thereal space point corresponding to the combination (pix#n, d) toward alens area unit (i, j) of the ith from the left and jth from above fromamong the Lx×Ly lens area units of the lens area (FIG. 30) is registeredas depicted in FIG. 32.

Further, where values of R, G and B as a luminance are allocated to aray heading from the real space point corresponding to the combination(pix#n, d) toward the lens area unit (i, j), into the pixel value tablefor the real space point corresponding to the combination (pix#n, d),the values of R, G and B as a luminance are registered.

FIG. 33 is a flow chart illustrating an example of the incident rayreproduction process performed by the incident ray reproduction section36 of FIG. 27.

At step S101, the real space point selection section 101, the collisiondecision section 103 and the luminance allocation section 104 of theincident ray reproduction section 36 (FIG. 27) acquire parallax mapsfrom the parallax information generation section 31. Then, theprocessing advances to step S102.

At step S102, the real space point selection section 101 selects, fromamong the pixels of the reference image HD1, one pixel that has not beenselected as a noticed pixel as yet as a noticed pixel. Then, theprocessing advances to step S103.

At step S103, the real space point selection section 101 refers to theparallax maps (multilayer parallax map) from the parallax informationgeneration section 31 to select, from among the parallaxes registeredfor the noticed pixel, one parallax that has not been selected as anoticed parallax as yet as a noticed parallax. Then, the processingadvances to step S104.

At step S104, the real space point selection section 101 selects a realspace point (x, y, z)=(x0, y0, z0) corresponding to the noticed pixel ofthe noticed parallax (noticed pixel having the noticed parallax) as anoticed real space point, and supplies the noticed real space point tothe ray generation section 102. Then, the processing advances to stepS105.

At step S105, the ray generation section 102 selects, from among thelens area units of the virtual lens (FIG. 30), one lens area unit thathas not been selected as a noticed lens area unit as yet as a noticedlens area unit. Then, the processing advances to step S106.

At step S106, the ray generation section 102 generates (a straight lineexpression of) a ray heading from the noticed real space point (x0, y0,z0) toward the center point (1x, 1y, 0) of the noticed lens area unit asa noticed ray and supplies the noticed ray to the collision decisionsection 103. Then, the processing advances to step S107.

Here, the straight line as a ray heading from the noticed real spacepoint (x0, y0, z0) toward the center point (1x, 1y, 0) of the noticedlens area unit is represented by an expression(x−1x)/(x0−1x)=(y−1y)/(y0−1y)=z/z0.

At step S107, the collision decision section 103 performs a collisiondecision taking the noticed ray from the ray generation section 102 as atarget. Then, the processing advances to step S108.

At step S108, the luminance allocation section 104 decides on the basisof a decision result of the collision decision (collision decisionresult) by the collision decision section 103 whether or not the noticedray collides.

If it is decided at step S108 that the noticed ray does not collide,namely, if, in the collision decision at step S107 by the collisiondecision section 103, a parallax equal to the parallax corresponding tothe noticed real space point (noticed parallax) is allocated to thenoticed ray, then the processing advances to step S109.

At step S109, the luminance allocation section 104 performs rayluminance allocation for allocating a luminance to the noticed ray andsupplies the allocated luminance to the light condensing processingsection 38. Then, the processing advances to step S110.

On the other hand, if it is decided at step S108 that the noticed raycollides, namely, if, in the collision decision at step S107 by thecollision decision section 103, a parallax that is not equal to theparallax (noticed parallax) corresponding to the noticed real spacepoint is allocated to the noticed ray, then the processing skips stepS109 and advances to step S110.

Accordingly, when a noticed ray collides, the ray luminance allocationat step S109 is not performed for the noticed ray.

At step S110, the ray generation section 102 decides whether or not allof the lens area units of the virtual lens have been selected as anoticed lens area unit.

If it is decided at step S110 that all of the lens area units of thevirtual lens have not been selected as a noticed lens area unit, thenthe processing returns to step S105, and thereafter, similar processesare repeated.

On the other hand, at step S110, if it is decided that all of the lensarea units of the virtual lens have been selected as a noticed lens areaunit, then the processing advances to step S111.

At step S111, the real space point selection section 101 decides whetheror not all of the parallaxes registered for the noticed pixel in themultilayer parallax map have been selected as a noticed parallax.

At step S111, if it is decided that all of the parallaxes registered forthe noticed pixel in the multilayer parallax map have not been selectedas a noticed parallax as yet, then the process returns to step S103, andthereafter, similar processes are repeated.

On the other hand, if it is decided at step S111 that all of theparallaxes registered for the noticed pixel in the multilayer parallaxmap have been selected as a noticed parallax, then the processingadvances to step S112.

At step S112, the real space point selection section 101 decides whetheror not all of the pixels of the reference image HD1 have been selectedas a noticed pixel.

If it is decided at step S112 that all of the pixels of the referenceimage HD1 have not been selected as a noticed pixel as yet, then theprocessing returns to step S102, and thereafter, similar processes arerepeated.

On the other hand, if it is decided at step S112 that all of the pixelsof the reference image HD1 have been selected as a noticed pixel, thenthe incident ray reproduction process is ended.

FIG. 34 is a flow chart illustrating an example of the process forcollision decision at step S107 of FIG. 33.

At step S121, the collision decision section 103 sets the parallax d forcollision decision to the maximum value Dmax as an initial value. Then,the processing advances to step S122.

At step S122, the collision decision section 103 determines the depth(distance) Z corresponding to the decision parallax d. Then, theprocessing advances to step S123.

At step S123, the collision decision section 103 decides whether or notthe depth Z corresponding to the decision parallax d is equal to thedepth z0 of the noticed real space point (x0, y0, z0).

If it is decided at step S123 that the depth Z corresponding to thedecision parallax d is not equal to the depth z0 of the noticed realspace point (x0, y0, z0), then the processing advances to step S124.

At steps beginning with step S124, it is confirmed whether or not thenoticed ray heading from the noticed real space point (x0, y0, z0)toward the center point (1x, 1y, 0) of the noticed lens area unitcollides with an object at the depth Z corresponding to the decisionparallax d.

In particular, at step S124, the collision decision section 103 sets aplane represented by an expression z=Z, namely, a plane perpendicular tothe optical axis at the position of the depth Z, as a decision layer asa plane for collision decision. Then, the processing advances to S125.

Here, the plane represented by the expression z=Z, which isperpendicular to the optical axis at the depth (distance) Zcorresponding to the parallax d, is hereinafter referred to also asparallax plane of z=Z. At step S124, the parallax plane of z=Z is set asa decision layer.

At step S125, the collision decision section 103 determines a crosspoint (Px, Py, Z) between the noticed ray and the decision layer. Then,the processing advances to step S126.

Here, the straight line as the noticed ray is represented by(x−1x)/(x0−1x)=(y−1y)/(y0−1y)=z/z0 as described hereinabove withreference to FIG. 33.

Accordingly, the x coordinate and the y coordinate of the noticed rayare represented by an expression x=z/z0(x0−1x)+1x and another expressiony=z/z0(y0−1y)+1y, respectively.

By substituting Z into the expression x=z/z0(x0−1x)+1x and theexpression y=z/z0(y0−1y)+1y, the x coordinate and the y coordinate ofthe noticed ray on the decision layer represented by the expression z=Z,namely, the x coordinate Px and the y coordinate Py of the cross point(Px, Py, Z) between the noticed ray and the decision layer, can bedetermined.

Accordingly, the x coordinate Px and the y coordinate Py can bedetermined in accordance with the expression x=Z/z0(x0−1x)+1x and theexpression y=Z/z0(y0−1y)+1y, respectively.

At step S126, the collision decision section 103 determines a pixel ofthe reference image corresponding to the cross point (Px, Py, Z) betweenthe noticed ray and the decision layer (the pixel is hereinafterreferred to also as cross point pixel). Then, the processing advances tostep S127.

At step s127, the collision decision section 103 decides whether or notthe decision parallax d is registered at (the position of) the crosspoint pixel in the multilayer parallax map (whether or not the parallaxregistered for the cross point pixel is equal to the decision parallaxd).

If it is decided at step S127 that the decision parallax d is notregistered at the cross point pixel, namely, if no object exists at thecross point (Px, Py, Z) between the noticed ray and the decision layerand the noticed ray does not collide with the cross point (Px, Py, Z),then the processing advances to step S128.

At step S128, the collision decision section 103 decrements (decreases)the decision parallax d by 1. Then, the processing returns to step S122,and thereafter, similar processes are repeated.

Here, as the decision parallax d is decreased at step S128, the decisionlayer moves from the position nearest to the virtual lens correspondingto the maximum value Dmax of the parallax toward the noticed real spacepoint (x0, y0, z0).

On the other hand, if it is decided at step S127 that the decisionparallax d is registered for the cross point pixel, namely, if an objectexists at the cross point (Px, Py, Z) between the noticed ray and thedecision layer and the noticed ray collides at the noticed point (Px,Py, Z), then the processing advances to step S129.

At step S129, the collision decision section 103 allocates the decisionparallax d to the noticed ray in order to represent the collisiondecision result that the noticed ray collides with an object before itreaches the virtual lens. Then, the processing returns.

On the other hand, if it is decided at step S123 that the depth Zcorresponding to the decision parallax d is equal to the depth z0 of thenoticed real space point (x0, y0, z0), namely, if the noticed ray doesnot collide with an object while the decision layer moves from theposition nearest to the virtual lens corresponding to the maximum valueDmax of the parallax to the noticed real space point (x0, y0, z0), thenthe processing advances to step S130.

At step S130, the collision decision section 103 allocates, in order torepresent the collision decision result that the noticed ray does notcollide with an object before it reaches the virtual lens, the noticedparallax (this also is the decision parallax d at this point of time),namely, the parallax d corresponding to the noticed real space point(x0, y0, z0) to the noticed ray. Thereafter, the processing returns.

It is to be noted that, while, in FIG. 34, the maximum value Dmax of theparallax is used as an initial value and the decision parallax issuccessively decremented from the initial value to a parallax of a goalthat corresponds to the noticed real space point (x0, y0, z0), the valueof the decision parallax may be changed in any manner over the range ofthe maximum value Dmax of the parallax to the parallax corresponding tothe noticed real space point (x0, y0, z0).

FIG. 35 is a flow chart illustrating an example of the process for rayluminance allocation at step S109 of FIG. 33.

At step S131, the luminance allocation section 104 detects acorresponding pixel corresponding to the noticed real space point (x0,y0, z0) from each of the picked up images HD1 to HD7 of the seven visualpoints. Then, the processing advances to step S132.

At step S132, the luminance allocation section 104 refers, for example,to the parallax maps of the picked up images HD1 to HD7 to detect, fromamong the corresponding pixels of the picked up images HD1 to HD7, theparallax d corresponding to the depth z0 of the noticed real space point(x0, y0, z0), namely, a pixel for which a parallax coincident with thetarget parallax is registered, as a ray luminance allocation pixel to beused for allocation of a luminance. Then, the processing advances tostep S133.

At step S133, the luminance allocation section 104 allocates, forexample, an average value of the pixel values (values of R, G and B) ofthe ray luminance allocation pixel as a luminance of a ray to thenoticed ray. Then, the processing returns.

In this manner, in the incident ray reproduction section 36 (FIG. 27),the ray generation section 102 generates a straight line as a ray thatis incident to the virtual lens, which has a synthetic aperture providedby the camera unit 21 ₁ to 21 ₇ of the seven visual points configuringthe image pickup apparatus 11, from a real space point with regard towhich a parallax is registered in the multilayer map, namely, from apoint on an object existing in a real space whose image pickup is to beperformed by the image pickup apparatus 11. In other words, the incidentray reproduction section 36 determines a straight line that describes,as a ray incident to the virtual lens from the real space point, the raygeometrically.

Further, in the incident ray reproduction section 36, the collisiondecision section 103 performs a collision decision for deciding whetheror not a ray collides with an object before it enters the virtual lens.

Then, in the incident ray reproduction section 36, the luminanceallocation section 104 allocates a luminance to rays that remain as aresult of the collision decision using the picked up images HD1 to HD7of the seven visual points picked up by the camera units 21 ₁ to 21 ₇.

Accordingly, it is possible to use the picked up images HD1 to HD7 ofthe seven visual points to reproduce a ray group incident to the virtuallens and hence to the emulation lens.

In other words, by making, for example, a so-called front lens thatconfigures the emulation lens correspond to the virtual lens, a raygroup incident to the virtual lens becomes a ray group that enters theemulation lens. Accordingly, by reproducing a ray group to be incidentto the virtual lens, a ray group that is to enter the emulation lens canbe reproduced.

As a result, a blur degree originating from that a ray group incident tothe emulation lens is condensed by the emulation lens can be reproducedby a light condensing process hereinafter described.

<Generation of Lens Information>

FIG. 36 is a view illustrating lens information generated by theemulation lens information generation section 37 of FIG. 3.

As the lens information (emulation lens information), a PSF intensitydistribution, an image plane pitch, PSF angle component information andimage plane shift information are available.

The PSF intensity distribution represents a response of the emulationlens to rays emitted from a point light source.

The image plane pitch represents a scale of the PSF intensitydistribution.

The PSF angle component information represents a position of the PSFintensity distribution when a ray emitted from a point light sourcereaches through the emulation lens.

The image plane shift information represents the image plane shiftposition that is a position on the virtual sensor that is reached, fromamong rays emitted from a real space point, by a principal ray throughthe emulation lens.

FIG. 37 is a view illustrating a real space point and a focus positionthat become a target for generation of lens information.

In particular, FIG. 37 is a schematic side elevational view when a realspace whose image is to be picked up by a camera unit 21 _(i)configuring the image pickup apparatus 11 as the virtual lens is viewedfrom the right with respect to the front of the image pickup apparatus.

The lens information is information that defines rays that pass theemulation lens, and the rays are emitted from a real space point.

Further, although the rays that pass the emulation lens are condensed onthe virtual sensor, the manner of condensing of rays differs dependingupon the focus position (focal length) f of the emulation lens.

Accordingly, the lens information can be generated as the number ofpieces of information equal to a maximum number of real space points tobe handled at most by the image processing apparatus 12 (FIG. 3) (thenumber is hereinafter referred to as maximum real space point number)for each focus position f of the emulation lens.

Now, it is assumed that the reference image HD1 is configured from Npixels pix1, pix2, . . . , pix#N and the number of parallaxes d that canbe registered into a parallax map (multilayer parallax map) is anintegral value of DPN=Dmax−Dmin+1 in one pixel increments from theminimum value Dmin to the maximum value Dmax as described hereinabovewith reference to FIG. 32.

In this case, the maximum real space point number is N×DPN.

Further, it is assumed now that the focus position f of the emulationlens can assume Fmax positions of f1, f2, . . . , f#Fmax.

In this case, the lens information can be formed as information ofFmax×N×DPN pieces at most.

Here, in the present embodiment, the size (scale) of the virtual sensoris defined on the basis of the emulation lens.

For example, where the emulation lens is an optical lens for an imagesensor of the 35 mm full size, the size of the virtual sensor is set tothe 35 mm full size on the basis of such an emulation lens as justdescribed. In particular, the horizontal and vertical sizes of thevirtual sensor are set, for example, to 36 mm and 24 mm, respectively.

Furthermore, in the present embodiment, the pixel pitch of the virtualsensor is defined on the basis of the number of pixels (resolution) of areference image such that the virtual sensor has the number of pixelsequal to the number of pixels of the reference image (or the number ofpixels smaller than the number of pixels of the reference image).

For example, where the emulation lens is an optical lens for an imagesensor of the 35 mm full size and the number of pixels of the referenceimage in the horizontal direction is Nx, since the horizontal size ofthe virtual sensor is 36 mm as described above, the pixel pitch of thevirtual sensor is 36 mm/Nx.

It is to be noted that a certain real space point (X, Y, Z) correspondsto a certain pixel of a reference image having a parallax d=Dcorresponding to the depth z=Z.

Further, since the pixel pitch of the virtual sensor is defined on thebasis of the number of pixels (resolution) of the reference image, thepixels of the virtual sensor can be made correspond to the pixels of thereference image. Where the virtual sensor has the number of pixels equalto the number of pixels of the reference image, a pixel of the referenceimage and a pixel of the virtual sensor positioned at the same positionas that of the pixel correspond to each other.

Further, if the parallax of a certain pixel of the reference image isadopted as it is as the parallax of a pixel of the virtual sensorcorresponding to the pixel, then the real space point (X, Y, Z)corresponds to a certain pixel of the virtual sensor having a parallaxd=D corresponding to the depth z=Z.

In this case, the lens information in regard to a certain real spacepoint can be considered lens information regarding a combination of apixel (position) and a parallax of the virtual sensor corresponding tothe real space point.

FIG. 38 is a view depicting an example of a PSF intensity distributionof a certain optical lens.

Referring to FIG. 38, the horizontal direction represents the focusposition f of the optical lens and the vertical direction indicates animage height that is a distance of an image formation position on animage formation plane, on which light from the optical lens forms animage, from the optical center.

Here, the image formation plane corresponds to the plane of the virtualsensor. Meanwhile, regarding the horizontal direction of FIG. 38, theleftward direction represents the focus position f near to the opticallens and the rightward direction represents the focus position f farfrom the optical lens. Furthermore, regarding the vertical direction ofFIG. 38, the upward direction represents a small image height and thedownward direction represents a great image height.

As depicted in FIG. 38, the PSF intensity distribution differs dependingupon the focus position f of the optical lens.

Further, the PSF intensity distribution differs depending upon the imageheight of the image formation position, namely, upon the position on theplane of the virtual sensor.

Further, the PSF intensity distribution differs also depending upon thedistance from the principal point of the optical lens to the imagepickup object (here, a point light source), namely, upon the parallax ofthe image pickup object.

Accordingly, the PSF intensity distribution differs depending, forexample, upon a set of the focus position f, (the position of) the pixelof the virtual sensor and the parallax of the image pickup object.

FIG. 39 is a view illustrating an example of a method for generating aPSF intensity distribution.

FIG. 39 depicts an outline where a real space whose image is picked upby a camera unit 21 _(i) configuring the image pickup apparatus 11 asthe virtual lens is viewed from the right with respect to the front ofthe image pickup apparatus similarly as in FIG. 37.

The emulation lens information generation section 37 generates a PSFintensity distribution for a real space point corresponding to each of amaximum real space point number N×DPN of real space points, namely, foreach of combinations of a maximum number N of pixels configuring thevirtual sensor and DPN parallaxes d that can be registered into amultilayer parallax map for each of the Fmax focus positions f at most.

Here, the maximum number N of the pixels configuring the virtual sensoris equal to the number N of the pixels pix1 to pix#N configuring thereference image HD1. As described hereinabove with reference to FIG. 37,in the present embodiment, the virtual sensor is configured from Npixels of the pixels pix1 to pix#N similarly to the reference image HD1in order to simplify the description.

The emulation lens information generation section 37 sets a point lightsource to a real space point and performs ray tracing for tracing a rayemitted from the point light source set to the real space point usinglens design data of the emulation lens to generate a PSF intensitydistribution.

In the ray tracing, setting a ray emitted from the point light source asan incident vector, a cross point between the incident vector and arefractive surface at the most image pickup object side of the emulationlens is calculated, and a vector when the incident vector as a rayincident from the cross point is refracted by and emitted from therefractive surface is calculated as an outgoing vector.

Further, in the ray tracing, using the outgoing vector as an incidentvector to a next refractive surface, a cross point between the incidentvector and the next refractive surface is calculated.

In the ray tracing, such processes as described above are repeated up tothe last refractive surface of the emulation lens.

Then, the emulation lens information generation section 37 observes theoutgoing vector emitted from the last refractive surface of theemulation lens on the virtual sensor and records the light intensity ofthe ray as the outgoing vector obtained as a result of the observationto generate a PSF intensity distribution.

It is assumed now that, in the emulation lens information generationsection 37, a rectangular area having a center (center of gravity) at aposition of the virtual sensor when a principal ray emitted from (thepoint light source of) the real space point, namely, a principal raythat is a ray that passes the principal point O of the emulation lensfrom among rays emitted from the real space point, reaches the virtualsensor is referred to as distribution area.

As the distribution area, for example, a rectangular area that iscentered at a principal ray emitted from the real space point and is aminimum (or close to the minimum) rectangular area that surrounds pointson the virtual sensor reached by rays emitted from the real space pointthrough the emulation lens can be adopted. Further, it is assumed thatthe distribution area is an area into which information can be recordedwith a resolution of PX×PY by width×length. For PX and PY, for example,255 can be adopted.

The emulation lens information generation section 37 records lightintensities, which become a PSF intensity distribution, in theresolution of PX×PY by width×length into the distribution area togenerate a PSF intensity distribution.

FIG. 40 is a view schematically depicting a PSF intensity distributiongenerated by the emulation lens information generation section 37.

As described hereinabove with reference to FIG. 39, the PSF intensitydistribution is generated for a real space point corresponding to eachof N×DPN combinations of N pixels pix1 to pix#N configuring the virtualsensor and DPN parallaxes d that can be registered into a parallax mapfor each of the Fmax focus positions f at most.

It is assumed now that a table in which the horizontal directionindicates DPN parallaxes d while the vertical direction indicates Npixels pix1 to pix#N configuring the virtual sensor and in which a PSFintensity distribution for a real space point corresponding to acombination of a certain parallax d and a certain pixel pix#n isregistered is referred to as intensity distribution table for the focusposition f, as illustrated in FIG. 40.

The emulation lens information generation section 37 generates anintensity distribution table for each of the Fmax focus positions f.

The PSF intensity distribution registered in the intensity distributiontable is recorded in a resolution of PX×PY by width×length in thedistribution area as described hereinabove with reference to FIG. 39.

Accordingly, when PSF intensity distributions are recorded into arrays,the number of arrays of the PSF intensity distributions isFmax×N×DPN×PX×PY at most.

Here, in the distribution area, a unit in which one light intensity of aPSF intensity distribution is recorded (unit of sampling of a PSFintensity distribution) is referred to as distribution area unit. Thedistribution area unit can be conceived, for example, as an area of asquare shape.

Since the distribution area is a minimum square area surrounding a pointon the virtual sensor reached by a ray emitted from a real space pointas described hereinabove with reference to FIG. 39, it has a variablesize.

Further, as described hereinabove with reference to FIG. 38, the PSFintensity distribution differs depending upon the focus position f,image height of the image formation position (distance between the realspace point (point light source) and the optical axis) and the distance(parallax) to the image pickup object (real space point (point lightsource)).

Also the size (scale) of the minimum distribution area surrounding sucha PSF intensity distribution as described above differs for each PSFintensity distribution.

In the light condensing process, as will be described later, anemulation image is generated by adding image formation values of raysthat form an image on the virtual sensor according to the PSF intensitydistributions recorded in distribution areas of different sizes.

Upon addition of image formation values according to PSF intensitydistributions, it is necessary to make the scale of distributions ofimage formation values according to PSF intensity distributions recordedin distribution areas of different sizes coincide with the scale of thevirtual sensor. Further, to this end, information representative of thescale of the PSF intensity distributions is required.

Therefore, the emulation lens information generation section 37determines an image plane pitch that is a size (pitch) of distributionarea units configuring a distribution area in which a PSF intensitydistribution is recorded as information representative of the PSFintensity distribution.

If it is assumed now that the image plane pitch is IP and the pixelpitch of the virtual sensor is PP, then in the light condensing process,distributions of image formation values of rays determined from a PSFintensity distribution are reduced (or expanded) to IP/PP times andadded on the virtual sensor.

It is to be noted that a PSF intensity distribution can be recorded notin a variable size but in a resolution of PX×PY in a distribution areaof a fixed size.

Where the PSF intensity distribution is recorded in a distribution areanot of a variable size but of a fixed size, only one image plane pitchis required for the fixed size.

However, since it is necessary to adjust the fixed size of adistribution area to a PSF intensity distribution that is spread mostfrom the reaching position of a principal ray on the virtual sensor, theresolution of a PSF intensity distribution having a narrow distributionis degraded.

FIG. 41 is a view schematically depicting an image plane pitch generatedby the emulation lens information generation section 37.

The image plane pitch is generated for each one PSF intensitydistribution.

Now, it is assumed that a table in which the horizontal directionindicates DPN parallaxes d while the vertical direction indicates Npixels pix1 to pix#N configuring the virtual sensor and in which animage plane pitch of a PSF intensity distribution for a real space pointcorresponding to a combination of a certain parallax d and a certainpixel pix#n is registered, for example, as depicted in FIG. 41 isreferred to as image plane pitch table for the focus position f.

The emulation lens information generation section 37 generates an imageplane pitch table for each of Fmax focus positions f.

Accordingly, where image plane pitches are recorded into arrays, thenumber of arrays of image plane pitches is Fmax×N×DPN at most.

FIG. 42 is a view illustrating an example of a method for generating PSFangle component information.

In particular, FIG. 42 is a front elevational view when the emulationlens is viewed from the front (image pickup object side).

The emulation lens information generation section 37 performs, forexample, when the ray generation section 102 described hereinabove withreference to FIG. 30 generates rays, a process similar to that performedfor the virtual lens for the emulation lens.

In particular, the emulation lens information generation section 37sets, for example, an area including the front lens of the emulationlens as a lens area.

In FIG. 42, for example, a minimum rectangular area surrounding thefront lens of the emulation lens is set as a lens area.

The emulation lens information generation section 37 divides (theemulation lens surrounded by) the lens area into lens area units ofsmall regions. Then, the emulation lens information generation section37 considers a real space point as a point light source and determinesPSF angle component information representative of a position of a PSFintensity distribution reached by rays incident to the lens area unitsfrom the real space point as the point light source through theemulation lens.

In FIG. 42, the lens area is divided into totaling Lx×Ly lens area unitsincluding Lx lens area units in the horizontal direction and Ly lensarea units in the vertical direction similarly as in the case of thevirtual lens of FIG. 30.

Also in FIG. 42, a grid point that is a cross point between a straightline in the horizontal direction and another straight line in thevertical direction by which the lens area surrounding the emulation lensis divided into the lens area units is represented also as grid pointLP#i (i=1, 2, . . . , (Lx+1) (Ly+1)) similarly as in the case of FIG.30.

Here, it is assumed that, in the present embodiment, in order tosimplify the description, the diameter is coincident between (thesynthetic aperture of) the virtual lens and (the aperture of) the frontlens of the emulation lens and also the size of the lens area and thedivision number Lx×Ly of the lens area are coincident between thevirtual lens and the emulation lens.

It is to be noted that, since the light condensing process is performedusing rays incident to the emulation lens, only it is necessary for thevirtual lens to have a diameter equal to or greater than the diameter ofthe emulation lens.

Further, in order to make rays incident to the virtual lens and raysincident to the emulation lens correspond to each other, the lens areaunits of the virtual lens and the lens area units of the emulation lensare made coincide (in size) with each other.

FIG. 43 is a view illustrating an example of a method for generating PSFangle component information.

FIG. 43 depicts an outline where a real space whose image is picked upby a camera unit 21 _(i) configuring the image pickup apparatus 11 asthe virtual lens is viewed from the right with respect to the front ofthe image pickup apparatus similarly to FIG. 37.

The emulation lens information generation section 37 generates PSF anglecomponent information for a real space point corresponding to each of amaximum real space point number N×DPN of real space points, namely, foreach of combinations of a maximum number N of pixels configuring thevirtual sensor and DPN parallaxes d that can be registered into amultilayer parallax map for each of the Fmax focus positions f at most.

The emulation lens information generation section 37 determines areaching point AP#i at which a ray emitted from (the point light sourceof) the real space point and incident to the grid point LP#i of theemulation lens reaches the virtual sensor.

Then, the emulation lens information generation section 37 converts thereaching point AP#i of the virtual sensor into a point on (thedistribution area) of the PSF intensity distribution and determines aset (in position) of the distribution area reaching point AP#i(distribution area unit reached by the ray) obtained by the conversionand the grid point LP#i as PSF angle component information.

It is to be noted that a ray emitted form a real space point andincident to the emulation lens may not necessarily reach the virtualsensor. In other words, among rays incident to the emulation lens, suchrays as not reach the virtual sensor (are not received by the virtualsensor) as indicated by a broken line arrow mark in FIG. 43 exists.

FIG. 44 is a view illustrating details of PSF angle componentinformation.

The emulation lens information generation section 37 determines, forfour grid points LP#i1, LP#i2, LP#i3 and LP#i4 that are four vertices ofthe lens area unit of the emulation lens, distribution area reachingpoints AP#i1, AP#i2, AP#i3 and AP#i4 that are reaching points on the PSFintensity distribution reached by rays emitted from the real space pointand passing the grid points LP#i1, LP#i2, LP#i3 and LP#i4, respectively.

Then, the emulation lens information generation section 37 generates theset of the four grid points LP#i1, LP#i2, LP#i3 and LP#i4 that are thefour vertices of the lens area unit of the emulation lens and thedistribution area reaching points AP#i1, AP#i2, AP#i3 and AP#i4 as PSFangle component information representative of an area (position) of aPSF intensity distribution reached by rays that pass the lens area unitwhose vertices are the four grid points LP#i1, LP#i2, LP#i3 and LP#i4.

After all, the PSF angle component information is a set of adistribution area reaching point AP#i and a grid point LP#i when a rayincident to the grid point LP#i of the emulation lens reaches thedistribution area reaching point AP#i through the emulation lens.

Here, the area of the PSF intensity distribution reached by a raypassing the lens area unit is referred to also as corresponding area.

In FIG. 44, the corresponding area is a quadrangular region havingvertices at the distribution area reaching points AP#i1, AP#i2, AP#i3and AP#i4.

The granularity (resolution) of the distribution area reaching pointsAP#i is a size of the distribution area unit (FIG. 40) of thedistribution area in which a PSF intensity distribution is recorded. Inparticular, the distribution area reaching points AP#i represent aposition of a certain distribution area unit of a distribution area.

FIG. 45 is a view schematically depicting PSF angle componentinformation generated by the emulation lens information generationsection 37.

The PSF angle component information is generated for real space pointscorresponding to N×DPN combinations of the N pixels pix1 to pix#Nconfiguring the virtual sensor and DPN parallaxes d that can beregistered into a parallax map for each of the Fmax focus positions f atmost.

Now, a table in which the horizontal direction indicates DPN parallaxesd while the vertical direction indicates N pixels pix1 to pix#Nconfiguring the virtual sensor and in which PSF angle componentinformation for a real space point corresponding to a combination of acertain parallax d and a certain pixel pix#n is registered, for example,as depicted in FIG. 45 is referred to as PSF angle component informationtable for the focus position f.

The emulation lens information generation section 37 generates a PSFangle component information table for each of the Fmax focus positionsf.

The PSF angle component information registered in the PSF anglecomponent information table is a set of a grid point LP#i of theemulation lens and a distribution area reaching point AP#i on thedistribution area of the PSF intensity distribution reached by a rayincident to the grid point LP#i through the emulation lens.

In the present embodiment, since the lens area of the emulation lens isdivided into PX×PY lens area units as described hereinabove withreference to FIG. 42, the number of grid points LP#i is (Lx+1) (Ly+1).

Accordingly, where the PSF angle component information is recorded intoarrays, the number of arrays of the PSF angle component information isFmax×N×DPN×(PX+1)×(PY+1) at most.

FIG. 46 is a view illustrating image plane shift information.

FIG. 46 depicts an outline when a real space whose image is picked up bythe camera unit 21 _(i) configuring the image pickup apparatus 11 as thevirtual lens is viewed from the right with respect to the front of theimage pickup apparatus similarly to FIG. 37.

In the present embodiment, the virtual lens and the front lens of theemulation lens are made correspond to each other, and the incident rayreproduction section 36 reproduces rays incident to the virtual lens asrays incident to the emulation lens.

However, since an emulation lens generally has a plurality of lenses,the virtual lens and the emulation lens are displaced in position (inthe z direction) of the entrance pupil.

Therefore, on the virtual sensor, an image observed through the virtuallens and an image observed through the emulation lens sometimes differfrom each other.

A of FIG. 46 is a view illustrating an example of an image observedthrough the virtual lens on the virtual sensor.

In A of FIG. 46, a straight line as a principal ray emitted from anobject obj1 in a real space and another straight line as a principal rayemitted from another object obj2 positioned at the front side withrespect to the object obj2 overlap with each other on the virtual lens.

Therefore, an image plane shift position of the object obj1 with respectto the virtual lens, namely, an image plane shift position that is aposition on the virtual sensor reached by a principal ray emitted fromthe object obj1 through the virtual lens, and an image plane shiftposition of the object obj2 with respect to the virtual lens coincidewith each other.

As a result, although the object obj2 is observed on the virtual sensor,the object obj1 positioned on the interior side than the object obj2 ishidden by the object obj2 and is not observed.

B of FIG. 46 is a view illustrating an example of an image observedthrough the emulation lens on the virtual sensor.

In B of FIG. 46, the objects obj1 and obj2 are positioned at the samepositions as those in the case of A of FIG. 46.

In the emulation lens, the position of the entrance pupil and hence theprincipal point are displaced from those of the virtual lens. Therefore,on the emulation lens, a straight line as a principal ray emitted fromthe object obj1 in the real space and another straight line as aprincipal ray emitted from the object obj2 positioned on the front sidewith respect to the object obj2 do not overlap with other but aredisplaced from each other.

Accordingly, since an image plane shift position of the object obj1 withrespect to the emulation lens and an image plane shift position of theobject obj2 with respect to the emulation lens do not coincide with eachother, both the object obj2 and the object obj1 positioned at theinterior side with respect to the object obj2 are observed on thevirtual sensor.

As described above, the image plane shift position of a real space pointdiffers between the virtual lens and the emulation lens originating fromthe displacement in position between the entrance pupils of the virtuallens and the emulation lens.

Therefore, the emulation lens information generation section 37generates image plane shift information representative of an image planeshift position with respect to the emulation lens as one kind of lensinformation in order to accurately reproduce light condensing of theemulation lens.

Here, the image plane shift information can be regarded as informationfor correcting the displacement in position of the entrance pupils ofthe virtual lens and the emulation lens, and from such a point of viewas just described, the image plane shift information can be regarded asentrance pupil correction information.

FIG. 47 is a view illustrating an example of a method for generatingimage plane shift information.

FIG. 47 depicts an outline when a real space whose image is to be pickedup by a camera unit 21 _(i) configuring the image pickup apparatus 11 asthe virtual lens is viewed from the right with respect to the front ofthe image pickup apparatus similarly to FIG. 37.

The emulation lens information generation section 37 generates imageplane shift information for a real space point corresponding to each ofa maximum real space point number N×DPN of real space points, namely,for each of combinations of a maximum number N of pixels configuring thevirtual sensor and DPN parallaxes d that can be registered into themultilayer parallax map for each of the Fmax focus positions f at most.

The emulation lens information generation section 37 sets a reachingpoint at which a principal ray emitted from (a point light source of) areal space point and passing the principal point of the emulation lensreaches the virtual sensor as an image plane shift position anddetermines coordinates (distances) in the x-axis and y-axis directions,for example, from the center of the virtual sensor, which represents theimage plane shift position, as image plane shift information.

FIG. 48 is a view schematically depicting image plane shift informationgenerated by the emulation lens information generation section 37.

The image plane shift information is generated for real space pointscorresponding to N×DPN combinations of N pixels pix1 to pix#Nconfiguring the virtual sensor and DPN parallaxes d that can beregistered into a parallax map for each of Fmax focus positions f atmost.

Now, a table in which the horizontal direction indicates DPN parallaxesd while the vertical direction indicates N pixels pix1 to pix#Nconfiguring the virtual sensor and in which image plane shiftinformation for a real space point corresponding to a combination of acertain parallax d and a certain pixel pix#n is registered, for example,as depicted in FIG. 48 is referred to as image plane shift informationtable for the focus position f.

The emulation lens information generation section 37 generates an imageplane shift information table for each of the Fmax focus positions f.

Accordingly, where image plane shift information is recorded intoarrays, the number of arrays of the image plane shift information isFmax×N×DPN at most.

It is to be noted that the lens information of the emulation lens (PSFintensity distribution, image plane pitch, PSF angle componentinformation and image plane shift information) can be determined byperforming an arithmetic operation of ray tracing or the like using lensdesign data of the emulation lens, and can be determined, where theemulation lens is an existing optical lens, by actually measuring a rayusing the optical lens.

FIG. 49 is a block diagram depicting an example of a configuration ofthe emulation lens information generation section 37 (FIG. 3) forgenerating lens information.

Referring to FIG. 49, the emulation lens information generation section37 includes a real space point selection section 131, an informationcalculation section 132 and a focus position selection section 133.

The real space point selection section 131 refers to a multilayerparallax map supplied from the parallax information generation section31 (FIG. 3) to the emulation lens information generation section 37 toselect a noticed real space point from among a maximum real space pointnumber N×DPN of real space points corresponding to combinations of the Npixels pix1 to pix#N configuring the virtual sensor and the DPNparallaxes d that can be registered into the multilayer parallax map.

The information calculation section 132 uses lens design data suppliedfrom the lens design data acquisition section 34 (FIG. 3) to theemulation lens information generation section 37 to generate lensinformation for the noticed real space point selected by the real spacepoint selection section 131 and the noticed focus position f selected bythe focus position selection section 133 and supplies the generated lensinformation to the light condensing processing section 38.

The focus position selection section 133 selects a noticed focusposition from among the Fmax focus positions f.

FIG. 50 is a flow chart illustrating an example of an emulation lensinformation generation process performed by the emulation lensinformation generation section 37 of FIG. 49.

At step S141, the focus position selection section 133 selects a noticedfocus position from among the Fmax focus positions f. Then, theprocessing advances to step S142.

Here, in the present embodiment, in order to reduce the informationamount of the lens information, lens information is generated only forthe noticed focus position. Selection of a noticed focus position can beperformed, for example, in response to an operation of a user or thelike. Further, as the noticed focus position, for example, a defaultfocus position determined in advance can be selected.

It is to be noted that the lens information can be generated not onlyfor the noticed focus position but for each of the Fmax focus positionsf.

At step S142, the real space point selection section 131 acquires amultilayer parallax map supplied from the parallax informationgeneration section 31. Then, the processing advances to step S143.

At step S143, the real space point selection section 131 selects, fromamong the pixels of the virtual sensor, one of the pixels that have notbeen selected as a noticed pixel as a noticed pixel. Then, theprocessing advances to step S144.

At step S144, the real space point selection section 131 selects, fromamong the parallaxes of the noticed pixel registered in the multilayerparallax map from the parallax information generation section 31, one ofthe parallaxes that have not been selected as a noticed parallax as yetas a noticed parallax. Then, the processing advances to step S145.

At step S145, the real space point selection section 131 selects a realspace point corresponding to the noticed pixel having the noticedparallax as a noticed real space point. Then, the processing advances tostep S146.

At step S146, the information calculation section 132 determines a PSFintensity distribution, an image plane pitch, PSF angle componentinformation and image plane shift information, which are lensinformation for the noticed real space point, namely, for the set of thenoticed focus position, noticed pixel and noticed parallax, in such amanner as described hereinabove with reference to FIGS. 39 to 48. Then,the processing advances to step S147.

At step S147, the real space point selection section 131 decides whetheror not all of the parallaxes of the noticed pixel registered in themultilayer parallax map have been selected as a noticed parallax.

If it is decided at step S147 that all of the parallaxes of the noticedpixel registered in the multilayer parallax map have not been selectedas a noticed parallax as yet, then the processing returns to step S144,and thereafter, similar processes are repeated.

On the other hand, if it is decided at step S147 that all of theparallaxes of the noticed pixel registered in the multilayer parallaxmap have been selected as a noticed parallax, then the processingadvances to step S148.

At step S148, the real space point selection section 131 decides whetheror not all of the pixels of the virtual sensor have been selected as anoticed pixel.

If it is decided at step S148 that all of the pixels of the virtualsensor have not been selected as a noticed pixel as yet, then theprocessing returns to step S143, and thereafter, similar processes arerepeated.

On the other hand, if it is decided at step S148 that all of the pixelsof the virtual sensor have been selected as a noticed pixel, then theemulation lens information generation process is ended.

<Light Condensing Process>

FIG. 51 is a view illustrating an outline of the light condensingprocess performed by the light condensing processing section 38 of FIG.3.

FIG. 51 depicts an outline when a real space whose image is to be pickedup by a camera unit 21 _(i) configuring the image pickup apparatus 11 asthe virtual lens is viewed from the right with respect to the front ofthe image pickup apparatus similarly to FIG. 37.

The light condensing processing section 38 performs a process fordetermining, using lens information from the emulation lens informationgeneration section 37, an image formation value when rays remaining as aresult of a collision decision from among rays supplied from theincident ray reproduction section 36 form an image on the virtual sensorthrough the emulation lens and adding the image formation value on thevirtual sensor as a light condensing process.

FIG. 52 is a view illustrating an example of a process for determiningan image formation value from within the light condensing process.

It is to be noted that, in FIG. 52, in order to avoid the figure frombecoming complicated, the lens area of the emulation lens is dividedinto 5×5 lens area units.

As described hereinabove with reference to FIG. 43, a ray emitted from areal space point and incident to the emulation lens may not necessarilyreach the virtual sensor. In other words, among rays incident to theemulation lens, rays that reach the virtual sensor and rays that do notreach the virtual sensor exist.

Now, a region of lens area units to which, from among rays emitted froma real space point and incident to the emulation lens, rays that reachthe virtual sensor are incident is referred to as effective ray region.

In FIG. 52, 3×3 lens area units rp1 to rp9 at the center part from among5×5 lens area units in regard to rays emitted from a certain real spacepoint (x, y, z) are an effective ray region.

Further, in FIG. 52, distribution area reaching points reached by raysemitted from the real space point (x, y, z) and passing grid points p#iof the lens areas are distribution area unit q#i of the distributionareas in which a PSF intensity distribution is recorded.

The distribution area unit q#i that becomes a distribution area reachingpoint reached by a ray emitted from the real space point (x, y, z) andpassing the grid point p#i of the lens area can be recognized from thePSF angle component information.

In FIG. 52, a ray BL emitted from the real space point (x, y, z) andpassing the lens area unit rp1 reaches the corresponding area rq1 of(the distribution area that has recorded therein) the PSF intensitydistribution.

Here, the lens area unit rp1 is a lens area unit having vertices at thegrid points p1, p2, p5 and p6.

Further, in FIG. 52, rays emitted from the real space point (x, y, z)and passing the grid points p1, p2, p5 and p6 reach the distributionarea units q1, q2, q5 and q6 of the distribution area in which a PSFintensity distribution is recorded. A region having vertices at thedistribution area units q1, q2, q5 and q6 is the corresponding area rq1.

According to the PSF angle component information, it is recognized thatrays emitted from the real space point (x, y, z) and passing the gridpoints p1, p2, p5 and p6 reach the distribution area units q1, q2, q5and q6 of the distribution area in which PSF intensity distribution isrecorded, respectively. As a result, it is recognized that the ray BLpassing the lens unit rp1 having vertices at the grid points p1, p2, p5and p6 reaches the corresponding area rq1 having the vertices at thedistribution area units q1, q2, q5 and q6.

In FIG. 52, the corresponding area rq#j is a corresponding area of a rayemitted from the real space point (x, y, z) and passing the lens areaunit rp#j.

The light condensing processing section 38 specifies the correspondingarea rq1 to be reached by a ray BL emitted from the real space point (x,y, z) and passing the lens area unit rp1 using the PSF angle componentinformation.

Then, the light condensing processing section 38 determines the productof the luminance allocated to the ray BL and the PSF intensitydistribution in the corresponding area rq1, namely, the PSF intensitydistribution recorded in (the position of) the distribution area unitsconfiguring the corresponding area rq1, as an image formation value whenthe ray BL forms an image on the virtual sensor through the emulationlens.

As described above, since the image formation value of the ray BL is theproduct of the luminance allocated to the ray BL and the PSF intensitydistribution recorded in each of the distribution area units configuringthe corresponding area rq1, such image formation values have adistribution whose granularity is the distribution area unit.

The light condensing processing section 38 determines an image formationvalue similarly also with regard to a ray emitted from the real spacepoint (x, y, z) and passing a lens area unit other than the lens areaunit rp1.

It is to be noted that, from among rays emitted from the real spacepoint (x, y, z) and incident to the 5×5 lens area units, those rays thatare incident to a lens area unit that is not the effective ray region(such rays are hereinafter referred to also as ineffective rays) do notreach the virtual sensor. Therefore, a lens area unit to which suchineffective rays are incident does not have a corresponding area to bereached by the ineffective rays. Accordingly, the image formation valuecan be determined only with regard to rays that pass the lens area unitsrp1 to rp9 that form the effective ray region.

FIG. 53 is a view illustrating another example of the process fordetermining an image formation value in the light condensing process.

It is to be noted that, in FIG. 53, like elements to those of FIG. 52are denoted by like reference symbols.

In FIG. 53, rays emitted from a real space point (x, y, z) do not reachlens area units rp7 to rp9 from among 3×3 lens area units rp1 to rp9 ata central portion, which is an effective ray region, of 5×5 lens areaunits, because they are blocked by an object existing on this side withrespect to the real space point (x, y, z).

Therefore, in FIG. 53, (a distribution of) an image formation value isdetermined substantially only with regard to each of rays that pass,from among the lens area units rp1 to rp9 that form the effective rayregion, the lens area units rp1 to rp6 reached by rays emitted from thereal space point (x, y, z).

FIG. 54 is a view illustrating an example of a process for adding (adistribution of) image formation values on the virtual sensor fromwithin the light condensing process.

As described hereinabove with reference to FIG. 52, image formationvalues of rays emitted from the real space point (x, y, z) indicate adistribution whose granularity is a distribution area unit. Now, it isassumed that, in order to facilitate the description, the distributionof image formation values of rays emitted from the real space point (x,y, z) is recorded in the distribution area in which a PSF intensitydistribution used to determine the image formation values is recorded.In other words, it is assumed that image formation values of raysemitted from the real space point (x, y, z) are recorded in a unit of adistribution area unit in the distribution area in which the PSFintensity distribution used to determine the image formation values isrecorded.

The light condensing processing section 38 adjusts the scale of thedistribution area such that the scale of the distribution area in whichthe distribution of the image formation values of rays emitted from thereal space point (x, y, z) is recorded is made coincide with the scaleof the virtual sensor using an image plane pitch in regard to the realspace point (x, y, z).

In particular, if it is assumed that the image plane pitch is IP and thepixel pitch of the virtual sensor is PP, then the light condensingprocessing section 38 performs a process for reducing (or expanding) thedistribution area, in which the distribution of the image formationvalues of rays is recorded, to IP/PP times as adjustment of the scale ofthe distribution area.

Further, the light condensing processing section 38 performs positioningof the position at which rays emitted from the real space point (x, y,z) are condensed on the virtual sensor through the emulation lensdepending upon the image plane shift position represented by the imageplane shift information for the real space point (x, y, z).

In particular, the light condensing processing section 38 performspositioning between the distribution area after adjustment of the scale,in which the distribution of the image formation values of rays emittedfrom the real space point (x, y, z) are recorded, and the virtual sensorsuch that the center point CP of the distribution area and the imageplane shift position of the virtual sensor coincide with each other.

After the light condensing processing section 38 preforms adjustment ofthe scale of the distribution area in which the distribution of imageformation values is recorded and besides performs positioning betweenthe distribution area after the adjustment of the scale and the virtualsensor in such a manner as described above, it adds the image formationvalues distributed in the distribution area on the virtual sensor in aunit of a pixel of the virtual sensor.

It is to be noted that the adjustment of the scale of (the distributionarea that has recorded therein) the image formation values and thepositioning may be performed in any order or may be performed at thesame time.

FIG. 55 is a block diagram depicting an example of a configuration ofthe light condensing processing section 38 of FIG. 3.

Referring to FIG. 55, the light condensing processing section 38includes a real space point selection section 141, an image formationvalue calculation section 142, a scale adjustment section 143, an imageformation position recognition section 144 and an addition section 145.

The real space point selection section 141 refers to a multilayerparallax map supplied from the parallax information generation section31 (refer to FIG. 3) to the light condensing processing section 38 toselect a noticed real space point from among a maximum real space pointnumber N×DPN of real space points corresponding to combinations of Npixels pix1 to pix#N configuring the reference image HD1 and DPNparallaxes d that can be registered into the multilayer parallax map.

The image formation value calculation section 142 uses a PSF intensitydistribution and PSF angle component information from within lensinformation supplied from the emulation lens information generationsection 34 to the light condensing processing section 38 to determine adistribution area in which a distribution of image formation values ofrays emitted from the noticed real space point selected by the realspace point selection section 131 from among rays supplied from theincident ray reproduction section 36 to the light condensing processingsection 38 is recorded, and supplies the distribution area to the scaleadjustment section 143.

The scale adjustment section 143 uses an image plane pitch from withinthe lens information supplied from the emulation lens informationgeneration section 34 to the light condensing processing section 38 toadjust the scale of the distribution area in which the distribution ofthe image formation values supplied from the image formation valuecalculation section 142 is recorded and supplies the distribution areaof the adjusted scale to the image formation position recognitionsection 144.

The image formation position recognition section 144 recognizes, fromimage plane shift information from within the lens information suppliedfrom the emulation lens information generation section 34 to the lightcondensing processing section 38, an image plane shift position that isan image formation position on the virtual sensor on which the rayspassing through the emulation lens form an image, and supplies therecognized image plane shift position to the addition section 145together with the distribution area after adjustment of the scale fromthe scale adjustment section 143.

The addition section 145 has a memory as the virtual sensor builttherein and performs positioning of the distribution area after theadjustment of the scale from the image formation position recognitionsection 144 and the virtual sensor (recognition of the position on thevirtual sensor at which the image formation values are to be added)depending upon the image plane shift position from the image formationposition recognition section 144.

Further, the addition section 145 (cumulatively) adds the imageformation values recorded in the distribution area after the positioningwith the virtual sensor on the virtual sensor in a unit of a pixel ofthe virtual sensor.

Then, the addition section 145 supplies an image, in which pixel valuesare provided by results of the addition of the image formation valuesobtained on the virtual sensor, namely, on the memory, as an emulationimage to the display apparatus 13 (FIG. 1).

FIG. 56 is a flow chart illustrating an example of the light condensingprocess performed by the light condensing processing section 38 of FIG.55.

It is to be noted that, in the present embodiment, in order to reducethe information amount of lens information, lens information isgenerated only for a noticed focus position as described hereinabovewith reference to FIG. 50. Therefore, the light condensing process isperformed under the assumption that the focus position is set to anoticed focus position when lens information is generated.

However, the lens information can be generated in advance for each ofthe Fmax focus positions (FIG. 37). In this case, the light condensingprocessing section 38 sets a noticed focus position and can perform alight condensing process using lens information regarding the noticedfocus position.

Alternatively, after the light condensing processing section 38 sets anoticed focus position, the emulation lens information generationsection 37 can generate lens information in regard to the noticed focusposition.

At step S151, the real space point selection section 141 acquires amultilayer parallax map supplied from the parallax informationgeneration section 31. Thereafter, the processing advances to step S152.

At step S152, the real space point selection section 141 selects a pixelthat has not been selected as a noticed pixel as yet from among thepixels of the reference image HD1 as a noticed pixel. Thereafter, theprocessing advances to step S153.

At step S153, the real space point selection section 141 selects oneparallax that has not been selected as a noticed parallax as yet fromamong the parallaxes of the noticed pixel registered in the multilayerparallax map from the parallax information generation section 31 as anoticed pixel. Thereafter, the processing advances to step S154.

At step S154, the real space point selection section 141 selects a realspace point corresponding to the noticed pixel having the noticedparallax as a noticed real space point. Thereafter, the processingadvances to step S155.

At step S155, the image formation value calculation section 142 selectsone lens area unit that has not been selected as a noticed lens areaunit as yet from among the lens area units of the emulation lens as anoticed lens area unit. Thereafter, the processing advances to stepS156.

At step S156, the image formation value calculation section 142 acquiresa ray heading from the noticed real space point toward the noticed lensarea unit from among rays supplied from the incident ray reproductionsection 36 as a noticed ray. Thereafter, the processing advances to stepS157.

At step S157, the image formation value calculation section 142 decideswhether or not the noticed ray reaches the emulation lens from thenoticed real space point.

If it is decided at step S157 that the noticed ray reaches the emulationlens, namely, when the parallax allocated to the noticed ray (parallaxallocated by the incident ray reproduction process described hereinabovewith reference to FIGS. 33 to 35) is equal to the noticed parallax, thenthe processing advances to step S158.

At step S158, the image formation value calculation section 142, scaleadjustment section 143, image formation position recognition section 144and addition section 145 perform a ray addition process hereinafterdescribed for a noticed ray that reaches the simulation lens, namely, anoticed ray that remains as a result of the collision decision.Thereafter, the processing advances to step S159.

On the other hand, if it is decided at step S157 that the noticed raydoes not reach the emulation lens, namely, if the parallax allocated tothe noticed ray (parallax allocated by the incident ray reproductionprocess described hereinabove with reference to FIGS. 33 to 35) is notequal to the noticed parallax, then the processing skips step S158 andadvances to step S159. Accordingly, when the noticed ray does not reachthe simulation lens, the ray addition process is not performed for thenoticed ray.

At step S159, the image formation value calculation section 142 decideswhether or not all of the lens area units of the emulation lens havebeen selected as a noticed lens area unit.

If it is decided at step S159 that all of the lens area units of theemulation lens have not been selected as a noticed lens area unit, thenthe processing returns to step S155, and thereafter, similar processesarea repeated.

On the other hand, if it is decided at step S159 that all of the lensarea units of the emulation lens have been selected as a noticed lensarea unit, then the processing advances to step S160.

At step S160, the real space point selection section 141 decides whetheror not all of the parallaxes of the noticed pixel registered in themultilayer parallax map have been selected as a noticed parallax.

If it is decided at step S160 that all of the parallaxes of the noticedpixel registered in the multilayer parallax map have not been selectedas a noticed parallax as yet, then the processing returns to step S153,and thereafter, similar processes are repeated.

On the other hand, if it is decided at step S160 that all of theparallaxes of the noticed pixel registered in the multilayer parallaxmap have been selected as a noticed parallax, then the processingadvances to step S161.

At step S161, the real space point selection section 141 decides whetheror not all of the pixels of the reference image HD1 have been selectedas a noticed pixel.

If it is decided at step S161 that all of the pixels of the referenceimage HD1 have not been selected as a noticed pixel as yet, then theprocessing returns to step S152. Thereafter, similar processes arerepeated.

On the other hand, if it is decided at step S161 that all of the pixelsof the reference image HD1 haven been selected as a noticed pixel, thenthe addition section 145 supplies an image obtained by the processesdescribed above and having pixel values provided by addition results ofimage formation values on the virtual sensor as an emulation image tothe display apparatus 13 (FIG. 1), thereby ending the light condensingprocess.

FIG. 57 is a flow chart illustrating an example of the ray additionprocess performed at step S158 of FIG. 56.

At step S171, the image formation value calculation section 142determines, using (a distribution area having recorded therein) a PSFintensity distribution regarding (a noticed pixel and a noticed parallaxcorresponding to) a noticed real space point regarding a noticed focusposition f and PSF angle component information from the emulation lensinformation generation section 37 as described hereinabove withreference to FIGS. 52 and 53, a corresponding area that is a position onthe distribution area at which the PSF intensity distribution reached bythe noticed ray is recorded.

Further, as described hereinabove with reference to FIGS. 52 to 53, theimage formation value calculation section 142 determines the product ofthe PSF intensity distribution of the corresponding area and theluminance allocated to the noticed ray (luminance allocated by theincident ray reproduction process described hereinabove with referenceto FIGS. 33 to 35) as (a distribution of) an image formation value ofthe noticed ray.

Then, the image formation value calculation section 142 supplies thedistribution area in which the image formation value of the noticed rayis recorded to the scale adjustment section 143. Thereafter, theprocessing advances from step S171 to step S172.

At step S172, as described hereinabove with reference to FIG. 54, thescale adjustment section 143 reduces or expands the distribution area,in which the distribution of the image formation value from the imageformation value calculation section 142 is recorded, using an imageplane pitch regarding the noticed real space point regarding the noticedfocus position f from the emulation lens information generation section34 to adjust the scale of the distribution area to a scale coincidentwith the scale of the virtual sensor.

Further, the scale adjustment section 143 supplies the distribution areaafter the adjustment of the scale to the addition section 145 throughthe image formation position recognition section 144. Thereafter, theprocessing advances from step S172 to step S173.

At step S173, the image formation position recognition section 144recognizes the image plane shift position, which is an image formationposition on the virtual sensor at which the noticed ray forms an imagethrough the emulation lens, from the image plane shift informationregarding the noticed real space point regarding the noticed focusposition f from the emulation lens information generation section 34,and supplies the image plane shift position to the addition section 145.Thereafter, the processing advances to step S174.

At step S174, the addition section 145 performs positioning between thedistribution area after the adjustment of the scale obtained by thescale adjustment section 143 and the virtual sensor depending upon theimage plane shift position from the image formation position recognitionsection 144.

In particular, the addition section 145 performs positioning between thedistribution area after the adjustment of the scale and the virtualsensor such that the center point CP (FIG. 54) of the distribution areaafter the adjustment of the scale and the image plane shift position ofthe virtual sensor coincide with each other.

Then, the addition section 145 adds the image formation values recordedin the distribution area after the positioning with the virtual sensoron the virtual sensor in a unit of a pixel of the virtual sensor. Inparticular, the addition section 145 adds the storage values of thememory as the virtual sensor and the image formation values and rewritesthe storage values of the memory with the addition values obtained bythe addition. It is to be noted that the storage values of the memory asthe virtual sensor are initialized to zero when the light condensingprocess (FIG. 56) is started.

The ray addition process ends therewith, and the processing returns.

As described above, in the lens emulation section 35 (FIG. 3), theincident ray reproduction section 46 reproduces rays that remain as aresult of the collision decision and are to enter the virtual lens.

Further, the emulation lens information generation section 37 generateslens information, namely, a PSF intensity distribution, an image planepitch, PSF angle component information and image plane shiftinformation.

Further, the light condensing processing section 38 determines, as animage formation value when each of rays that remain as a result of thecollision decision forms an image on the virtual sensor through theemulation lens, the product of the PSF intensity distribution and theluminance of the ray at a position of the PSF intensity distributionrepresented by the PSF angle component information.

Further, the light condensing processing section 38 adjusts the scale ofthe distribution of the image formation values of the rays on the basisof the image plane pitch so as to make the scale coincident with thescale of the virtual sensor.

Then, the light condensing processing section 38 performs positioning ofthe position on the virtual sensor at which the image formation value isto be added depending upon the image plane shift position and performsaddition of the image formation value on the virtual sensor, and thengenerates an emulation image in which a pixel value is given by theaddition value obtained by the addition.

According to such processes of the lens emulation section 35 asdescribed above, light condensing equivalent to that by an actualoptical lens is reproduced by a digital signal process, and as a result,an emulation image that reproduces (reflects) a blur degree or otherlight condensing characteristics of an actual optical lens can begenerated.

Accordingly, even if the user does not purchase an actual optical lens,it can enjoy such a high-quality image pickup experience (experience ofimage pickup performed using a high-quality optical lens) that the userperforms image pickup using the optical lens.

<Reduction of Information Amount of Lens Information>

FIG. 58 is a view illustrating an outline of reduction of theinformation amount of lens information.

A of FIG. 58 depicts an example of a PSF intensity distribution fromwithin lens information of an actual optical lens.

In particular, A of FIG. 58 schematically depicts a relationship betweenimage formation positions on an actual image sensor at which rays forman image through the actual optical lens and a PSF intensitydistribution applied to the rays that form an image at the imageformation positions.

As described hereinabove with reference to FIG. 38, the PSF intensitydistribution differs depending upon the focus position f, the imageheight at the image formation position (distance from the optical axis)and the distance (parallax) to the image pickup object (real spacepoint).

In particular, for example, if the image height of the image formationposition on an actual image sensor (distance of a real space point thatforms an image at the image formation position from the optical axis)differs, then the PSF intensity distribution of the actual optical lensapplied to the ray emitted from the real space point that forms an imageat the image formation position differs.

Accordingly, in the actual image sensor, the PSF intensity distributionof the actual optical lens provides different infinite information ifthe image height of the image formation position differs.

Although the PSF intensity distribution differs if the image height ofthe image formation position differs in such a manner as describedabove, conversely speaking, where the image height is same, namely,where the real space point is positioned at the same distance from theoptical axis, the PSF intensity distribution is common unless the focusposition f and the parallax (distance to the real space point) vary.

In particular, in order to simplify the description, it is assumed nowthat the focus position f and the parallax are fixed.

The PSF intensity distribution in regard to a real space pointcorresponding to a position pos2 after a certain position pos1 of acertain image height r of an image sensor is rotated by a predeterminedangle a around the optical axis coincides with the PSF intensitydistribution after rotation when the PSF intensity distribution inregard to the real space point corresponding to the position pos1 isrotated by the predetermined angle a around the optical axis.

Since the PSF intensity distribution in regard to the real space pointcorresponding to the position pos2 coincides with the PSF intensitydistribution after rotation when the PSF intensity distribution inregard to the real space point corresponding to the position pos1 isrotated by the predetermined angle a around the optical axis asdescribed above, the image plane pitch in regard to the real space pointcorresponding to the position pos2 coincides with the image plane pitchin regard to the real space point corresponding to the position pos1.

Further, the PSF angle component information in regard to the real spacepoint corresponding to the position pos2 coincides with PSF anglecomponent information after rotation when the PSF angle componentinformation in regard to the real space point corresponding to theposition pos1 is rotated by the predetermined angle a around the opticalaxis.

Furthermore, (the image plane shift position represented by) the imageplane shift information in regard to the real space point correspondingto the position pos2 coincides with the image plane shift informationafter rotation when the image plane shift information in regard to thereal space point corresponding to the position pos1 is rotated by thepredetermined angle a around the optical axis.

As described above, where the focus position f and the parallax arefixed, the lens information is common among real space pointscorresponding to positions of the image sensor at which the image heightis same.

Therefore, the emulation lens information generation section 37 canreduce the information amount of lens information by generating lensinformation not for real space points corresponding to all pixels of thevirtual sensor but only for real space points corresponding to aplurality of information points that are plural positions of part of theplane of the virtual sensor.

In particular, the emulation lens information generation section 37determines, for example, (real space points of) a predetermined one axisextending in the plane of the virtual sensor from the center of thevirtual sensor (optical axis) as a lens information generation axis of atarget for generation of lens information and sets (real space pointscorresponding to) a plurality of positions on the lens informationgeneration axis as information points for generating lens information.

Then, the emulation lens information generation section 37 generateslens information regarding (the real space points corresponding to) theinformation points of the lens information generation axis.

B of FIG. 58 depicts an example of the lens information generation axis.

In B of FIG. 58, one axis extending in an upward direction from thecenter of the virtual sensor forms the lens information generation axis.

The lens information generated for the information points of such a lensinformation generation axis as described above can be applied, forexample, to a light condensing process for rays emitted from a realspace point corresponding to the position of the virtual sensorcoincident with the lens generation axis after rotation when rotation isperformed around the center of the virtual sensor such that the lensinformation generation axis is rotated by a rotational angle equal tothat of the rotation.

FIG. 59 is a view depicting a particular example of the lens informationgeneration axis.

Now, an axis heading from the center of the virtual sensor toward onepixel at a diagonal of the virtual sensor as depicted in FIG. 59 isreferred to as diagonal axis.

In FIG. 59, a plurality of positions such as 15 positions are set atequal distances as information positions on a diagonal axis.

Further, in FIG. 59, the diagonal axis to which the information pointsare set is rotated around the center of the virtual sensor such that itis directed in an upward direction, and the diagonal axis after therotation is the lens information generation axis.

Accordingly, in FIG. 59, the lens information generation axis is a linesegment that has a width of 0 and a vertical length equal to 1/2 thelength of the diagonal of the virtual sensor (distance between pixels atthe diagonal positions) and extends in the vertical direction (upwarddirection) from the center of the virtual sensor.

The emulation lens information generation section 37 can generate lensinformation only for real space points corresponding to the informationpoints of such a lens information generation axis as described above.The real space points corresponding to the information points of thelens information generation axis are points in the plane represented byx=0.

It is to be noted that, although 15 or a like number of informationpoints are sufficient in regard to the PSF intensity distribution, PSFangle component information and image plane pitch from within the lensinformation, for the image plane shift information, 15 or a like numberof information points sometimes degrade the reproducibility of lightcondensing characteristics of the emulation lens in the light condensingprocess.

Therefore, for the image plane shift information, a value obtained bydividing the distance from the center of the virtual sensor to one pixelpositioned at the diagonal (maximum value of the image height of thevirtual sensor) by the pixel pitch of the virtual sensor (valueproximate to 1/2 the number of pixels on the diagonal of the virtualsensor) or the like can be adopted as the number of information pointsto be provided on the lens information generation axis.

Here, if lens information is generated for real space pointscorresponding to N×DPN combinations of N pixels pix1 to pix#Nconfiguring the virtual sensor and DPN parallaxes d that can beregistered into a parallax map for each of Fmax focus positions f asdescribed hereinabove with reference to FIGS. 40, 41, 45 and 48, thenthe number of arrays of the lens information becomes such a huge numberas described below.

In particular, the number of arrays of the PSF intensity distribution isFmax×N×DPN×PX×PY at most as described hereinabove with reference to FIG.40. The number of arrays of the image plane pitch is Fmax×N×DPN at mostas described hereinabove with reference to FIG. 41. The number of arraysof the PSF angle component information is Fmax×N×DPN×(PX+1)×(PY+1) atmost as described hereinabove with reference to FIG. 45. The number ofarrays of image plane shift information is Fmax×N×DPN at most asdescribed hereinabove with reference to FIG. 48.

It is to be noted that PX and PY represent the horizontal (horizontaldirection) and vertical (vertical direction) numbers of lens area unitsconfiguring a lens area, respectively, as described hereinabove withreference to FIG. 42.

On the other hand, if the number of information points on the lensinformation generation axis is represented as Ninfo, then the number ofarrays of the lens information is such as described below.

In particular, the number of arrays of the PSF intensity distribution isFmax×Ninfo×DPN×PX×PY at most. The number of arrays of the image planepitch is Fmax×Ninfo×DPN at most. The number of arrays of the PSF anglecomponent information is Fmax×Ninfo×DPN×(PX+1)×(PY+1) at most. Thenumber of arrays of the image plane shift information is Fmax×Ninfo×DPNat most.

Accordingly, when lens information is generated only for the informationpoints of the lens information generation axis, the information amountof the lens information can be reduced to Ninfo/N in comparison withthat where lens information is generated for real space pointscorresponding to N×DPN combinations of N pixels pix1 to pix#Nconfiguring the virtual sensor and DPN parallaxes d that can beregistered into a parallax map for each of Fmax focus positions f.

For example, if the number of pixels N of the virtual sensor is1892×1052 and the number Ninfo of the information points is 15, theinformation amount of the lens information can be reduced to15/(1892×1052).

As a result, with the lens information generation axis, a blur degree orother light condensing characteristics of the emulation lens can bereproduced by a reduced data amount.

FIG. 60 is a block diagram depicting an example of a configuration ofthe emulation lens information generation section 37 of FIG. 3 wherelens information is generated only for the information points of thelens information generation axis.

It is to be noted that, in FIG. 60, like elements to those in FIG. 49are denoted by like reference numerals, and description of them issuitably omitted in the following description.

Referring to FIG. 60, the emulation lens information generation section37 includes a real space point selection section 231, an informationcalculation section 132 and a focus position selection section 133.

Accordingly, the emulation lens information generation section 37 ofFIG. 60 is common to the case of FIG. 49 in that it includes theinformation calculation section 132 and the focus position selectionsection 133.

However, the emulation lens information generation section 37 of FIG. 60is different from the case of FIG. 49 in that it includes the real spacepoint selection section 231 in place of the real space point selectionsection 131.

The real space point selection section 231 refers to a multilayerparallax map supplied from the parallax information generation section31 (FIG. 3) to select a noticed real space point from among Ninfo×DPNreal space points corresponding to combinations of Ninfo informationpoints of a lens information generation axis on the virtual sensor andDPN parallaxes d that can be registered into the multilayer parallaxmap.

FIG. 61 is a flow chart illustrating an example of an emulation lensinformation generation process performed by the emulation lensinformation generation section 37 of FIG. 60.

At step S211, the focus position selection section 133 selects a noticedfocus position from among the Fmax focus positions f similarly as atstep S141 of FIG. 50. Thereafter, the processing advances to step S212.

At step S212, the real space point selection section 231 acquires amultilayer parallax map supplied from the parallax informationgeneration section 31 similarly as at step S142 of FIG. 50. Thereafter,the processing advances to step S213.

At step S213, the real space point selection section 231 sets a lensinformation generation axis to be used for generation of a PSF intensitydistribution, PSF angle component information and an image plane pitchfrom within the lens information. Thereafter, the processing advances tostep S214.

In particular, the real space point selection section 231 sets a lensinformation generation axis, which has a predetermined number of (forexample, 15 or the like) information points determined in advance andwhich equally divides, for example, the maximum image height of thevirtual sensor (distance from the center of the virtual sensor to onepixel on the diagonal), in a vertical direction (upward direction) froma start point set to the center of the virtual lens.

At step S214, the real space point selection section 231 selects, fromamong the information points of the lens information generation axis,one information point that has not been selected as a noticedinformation point as yet as a noticed information point. Thereafter, theprocessing advances to step S215.

At step S215, the real space point selection section 231 selects, fromamong parallaxes that are registered in the multilayer parallax map fromthe parallax information generation section 31 and can be registeredinto a pixel at the position of the noticed information point (pixelnear to the noticed information point), one parallax that has not beenselected as a noticed parallax as yet as a noticed parallax. Thereafter,the processing advances to step S216.

At step S216, the real space point selection section 231 selects a realspace point corresponding to the noticed information point having thenoticed parallax (position of the noticed information point on the planeof the virtual sensor) as a noticed real space point. Thereafter, theprocessing advances to step S217.

At step S217, the information calculation section 132 determines a PSFintensity distribution, an image plane pitch and PSF angle componentinformation for the noticed real space point, namely, for a set of thenoticed focus position, the noticed information point and the noticedparallax, in a similar manner as at step S146 of FIG. 50. Thereafter,the processing advances to step S218.

At step S218, the real space point selection section 231 decides whetheror not all of the parallaxes that can be registered into the multilayerparallax map have been selected as a noticed parallax.

If it is decided at step S218 that all of the parallaxes that can beregistered into the multilayer parallax map have not been selected as anoticed parallax as yet, then the processing returns to step S215, andthereafter, similar processes are repeated.

On the other hand, if it is decided at step S218 that all of theparallaxes that can be registered into the multilayer parallax map havebeen selected as a noticed parallax, then the processing advances tostep S219.

At step S219, the real space point selection section 231 decides whetheror not all of the information points of the lens information generationaxis have been selected as a noticed information point.

If it is decided at step S219 that all of the information points of thelens information generation axis have not been selected as a noticedinformation point as yet, then the processing returns to step S214, andthereafter, similar processes are repeated.

On the other hand, if it is decided at step S219 that all of theinformation points of the lens information generation axis have beenselected as a noticed information point, then the processing advances tostep S220, and thereafter, image plane shift information is generated.

At step S220, the real space point selection section 231 sets a lensinformation generation axis to be used for generation of image planeshift information from within the lens information. Thereafter, theprocessing advances to step S221.

In particular, the real space point selection section 231 sets a lensinformation generation axis, on which, for example, the number ofinformation points equal to the number of a value obtained by dividingthe maximum image height of the virtual sensor by the pixel pitch of thevirtual sensor are disposed at equal distances, in the verticaldirection from a start point set to the center of the virtual lens.

At step S221, the real space point selection section 231 selects, fromamong the information points of the lens information generation axis,one information point that has not been selected as a noticedinformation point as yet as a noticed information point. Thereafter, theprocessing advances to step S222.

At step S222, the real space point selection section 231 selects, fromamong parallaxes that are registered in the multilayer parallax map fromthe parallax information generation section 31 and can be registeredinto a pixel at the position of the noticed information point, oneparallax that has not been selected as a noticed parallax as yet as anoticed parallax. Thereafter, the processing advances to step S223.

At step S223, the real space point selection section 231 selects a realspace point corresponding to the noticed information point having thenoticed parallax as a noticed real space point. Thereafter, theprocessing advances to step S224.

At step S224, the information calculation section 132 determines imageplane shift information for the noticed real space point, namely, forthe set of the noticed focus position, the noticed information point andthe noticed parallax, similarly as at step S146 of FIG. 50. Thereafter,the processing advances to step S225.

At step S225, the real space point selection section 231 decides whetheror not all of the parallaxes that can be registered into the multilayerparallax map have been selected as a noticed parallax.

If it is decided at step S225 that all of the parallaxes that can beregistered into the multilayer parallax map have not been selected as anoticed parallax as yet, then the processing returns to step S222, andthereafter, similar processes are repeated.

On the other hand, if it is decided at step S225 that all of theparallaxes that can be registered into the multilayer parallax map havebeen selected as a noticed parallax, then the processing advances tostep S226.

At step S226, the real space point selection section 231 decides whetheror not all of the information points of the lens information generationaxis have been selected as a noticed information point.

If it is decided at step S226 that all of the information points of thelens information generation axis have not been selected as a noticedinformation point as yet, then the processing returns to step S221, andthereafter, similar processes are repeated.

On the other hand, if it is decided at step S226 that all of theinformation points of the lens information generation axis have beenselected as a noticed information point, then the emulation lensinformation generation process is ended.

In the emulation lens information generation process of FIG. 61, lensinformation regarding real space points corresponding to sets of theparallaxes that can be registered into the multilayer parallax map andinformation points of the lens information generation axis is determinedin such a manner as described above.

FIG. 62 is a view illustrating an example of a light condensing processperformed using lens information generated for (the real space pointscorresponding to) the information points of the lens informationgeneration axis in such a manner as described above.

As a method for performing a light condensing process using lensinformation generated for the information points of the lens informationgeneration axis, a method for rotating the lens information (hereinafterreferred to as lens information rotation method) and a method forrotating a ray to be used for a light condensing process (hereinafterreferred to as ray rotation method) are available.

Now, an angle when (a pixel of) the virtual sensor or the lensinformation generation axis is rotated around the center of the virtualsensor such that, for example, the pixel (position) of the virtualsensor corresponding to a certain real space point rsp is positioned onthe lens information generation axis is referred to as coincidencerotation angle ALPHA.

The pixel of the virtual sensor corresponding to the real space pointrsp and the lens information generation axis form the coincidencerotation angle ALPHA around the center of the virtual sensor.

In the lens information rotation method and the ray rotation method,when (a pixel of) the virtual sensor or the lens information generationaxis is rotated by the coincidence rotation angle ALPHA around thecenter of the virtual sensor such that the pixel of the virtual sensorcorresponding to the real space point rsp is positioned on the lensinformation generation axis, an information point nearest to the pixelof the virtual sensor corresponding to the real space point rsp(hereinafter referred to as corresponding information point) isdetected.

Here, the clockwise direction of the rotation angle upon rotation aroundthe center of the virtual sensor is determined as positive direction. Inthis case, by rotating the lens information generation axis by thecoincidence rotation angle +ALPHA around the center of the virtualsensor or by rotating the virtual sensor by the coincidence rotationangle −ALPHA around the center of the virtual sensor, the pixel of thevirtual sensor corresponding to the real space point rsp comes to bepositioned on the lens information generation axis.

In the following, in order to simplify the description, it is assumedthat, upon detection of a corresponding information point, from betweenthe lens information generation axis and the virtual sensor, forexample, the lens information generation axis is rotated by coincidencerotation angle +ALPHA.

In the lens information rotation method and the ray rotation method,when the lens information generation axis is rotated by the coincidencerotation angle +ALPHA around the center of the virtual sensor, aninformation point nearest to the pixel of the virtual sensorcorresponding to the real space point rsp is detected as a correspondinginformation point.

Then, a light condensing process is performed applying the lensinformation regarding (the real space point corresponding to) thecorresponding information point to a ray emitted from the real spacepoint rsp.

However, in the lens information rotation method, (the PSF intensitydistribution, the PSF angle component information and the image planeshift information from within) the lens information regarding thecorresponding information point is rotated by the coincidence rotationangle +ALPHA and applied to the ray emitted from the real space pointrsp.

Meanwhile, in the ray rotation method, a ray emitted from the real spacepoint rsp is rotated by the coincidence rotation angle −ALPHA, and lensinformation regarding the corresponding information point is applied tothe ray after the rotation.

FIG. 62 depicts an example of a light condensing process by the lensinformation rotation method.

A of FIG. 62 depicts an example of calculation of an image formationvalue in the light condensing method where the noticed pixel of thevirtual sensor corresponding to the noticed real space point is a pixelon the lens information generation axis (hereinafter referred to ason-axis pixel).

In particular, A of FIG. 62 depicts an example of (a distribution areahaving recorded therein) a PSF intensity distribution regarding acorresponding information point regarding a noticed pixel of the virtualsensor corresponding to a noticed real space point.

In A of FIG. 62, a ray incident to a lens area unit U1 reaches acorresponding area UC1 of a PSF intensity distribution.

Where the noticed pixel of the virtual sensor corresponding to thenoticed real space point is an on-axis pixel, for a ray emitted from thenoticed real space point and incident to the lens area unit U1, the PSFintensity distribution regarding the corresponding information point tothe noticed pixel is used as it is (without rotating the same), and theproduct of the luminance allocated to the ray incident to the lens areaunit U1 and the PSF intensity distribution of the corresponding area UC1is determined as an image formation value.

B of FIG. 62 depicts an example of calculation of an image formationvalue in a light condensing process where the noticed pixel of thevirtual sensor corresponding to the noticed real space point is a pixelon a straight line provided by the lens information generation axis whenthe lens information generation axis is rotated, for example, by 90degrees around the center of the virtual sensor (the pixel is referredto also as 90-degree rotation pixel).

In B of FIG. 62, the coincidence rotation angle +ALPHA of a 90-degreerotation pixel that is the noticed pixel is 90 degrees.

Now, it is assumed that the corresponding information point to the90-degree rotation pixel that is the noticed pixel coincides with thecorresponding information point to the on-axis pixel in the case of A ofFIG. 62.

In this case, according to the lens information rotation method, for aray emitted from the noticed real space point, an image formation valueis determined using a PSF intensity distribution after rotation when aPSF intensity distribution regarding the corresponding information pointto the noticed pixel is rotated, for example, by 90 degrees that is thecoincidence rotation angle +ALPHA around the center of the distributionarea in which the PSF intensity distribution is recorded.

For (a distribution area having recorded thereon) a PSF intensitydistribution after rotation, a ray incident to the lens area unit U2positioned at the position of the lens area unit U1 after rotation whenthe lens area unit U1 is rotated by 90 degrees that is the coincidencerotation angle +ALPHA around the optical axis reaches the correspondingarea UC1 of the PSF intensity distribution.

Accordingly, where the noticed pixel is a 90-degree rotation pixel, asan image formation value of a ray emitted from the noticed real spacepoint and incident to the lens area unit U2, the product of theluminance allocated to the ray incident to the lens area unit U2 and thePSF intensity distribution of the corresponding area UC1 is determinedin the lens information rotation method.

FIG. 63 depicts an example of a light condensing process according tothe ray rotation method.

Where the noticed pixel of the virtual sensor corresponding to thenoticed real space point is an on-axis pixel, in the ray rotationmethod, the image formation value of a ray emitted from the noticed realspace point is determined similarly as in the lens information rotationmethod.

On the other hand, where the noticed pixel of the virtual sensorcorresponding to the noticed real space point is, for example, a90-degree rotation pixel, in the ray rotation method, a ray afterrotation when a ray emitted from the noticed real space point is rotatedby coincidence rotation angle −ALPHA=−90 degrees is used to determine animage formation value of the ray.

A of FIG. 63 depicts rotation of a ray emitted from the noticed realspace point.

Where the noticed pixel of the virtual sensor corresponding to thenoticed real space point is a 90-degree rotation pixel, a ray to beincident to the emulation lens is rotated by the coincidence rotationangle −ALPHA=−90 degrees around the optical axis.

In A of FIG. 63, before rotation of the ray, a ray R1 is incident to thelens area unit U1 from the noticed real space point, and another ray R2is incident to the lens area unit U2 from the noticed real space point.

The lens area unit U2 is positioned at a position when the lens areaunit U1 is rotated by the coincidence rotation angle +ALPHA=+90 degreesaround the optical axis.

Accordingly, if a ray to be incident to the emulation lens is rotated bythe coincidence rotation angle −ALPHA=−90 degrees around the opticalaxis, the ray R2 that has been incident to the lens area unit U2 beforethe rotation now enters the lens area unit U1.

B of FIG. 63 depicts an example of (a distribution area having recordedtherein) a PSF intensity distribution regarding the correspondinginformation point regarding the 90-degree rotation pixel that is thenoticed pixel.

In B of FIG. 63, a ray incident to the lens area unit U1 reaches thecorresponding area UC1 of the PSF intensity distribution similarly as inthe case of A of FIG. 62.

Since the ray R2 after the rotation enters the lens area unit U1, theimage formation value VF of the ray R2 is determined as the product ofthe luminance of the ray R2 and the PSF intensity distribution of thecorresponding area UC1 to the lens area unit U1.

C of FIG. 63 depicts an example of reverse rotation of the imageformation value VF of the ray R2.

In the ray rotation method, when an image formation value VF is added tothe virtual sensor, (the distribution area having recorded thereon) theimage formation value VF is rotated reversely by the coincidencerotation angle −ALPHA=−90 degrees by which the ray has been rotatedaround the optical axis.

In particular, (the distribution area having recorded therein) the imageformation value VF is added on the virtual sensor after it is rotated bythe coincidence rotation angle +ALPHA=+90 degrees around the opticalaxis.

In the light condensing process performed using lens informationgenerated for the lens information generation axis, the lens informationrotation method may be adopted or the ray rotation method may beadopted.

According to the lens information rotation method, the arithmeticoperation amount required for the light condensing process can be madesmaller than that by the ray rotation method.

However, in the lens information rotation method, the reproducibility ofthe light condensing characteristics of the emulation lens may possiblybe degraded.

In particular, in the lens information rotation method, lens information(except the image plane pitch) is rotated by the coincidence rotationangle +ALPHA.

In this case, the coordinate system of the distribution area reachingpoint AP#i (FIG. 44) represented, for example, by the PSF anglecomponent information from within the lens information after therotation is a coordinate system inclined by the coincidence rotationangle +ALPHA. Therefore, except a case in which the coincidence rotationangle +ALPHA is an integer multiple of 90 degrees, a displacement occursbetween a position that may possibly become the distribution areareaching point AP#i represented by the PSF angle component informationbefore the rotation and a position that may possibly become thedistribution area reaching point AP#i represented by the PSF anglecomponent information after the rotation.

Since the distribution area reaching point AP#i represented by the PSFangle component information represents a position on the PSF intensitydistribution reached by a ray in a granularity of a distribution areaunit (FIG. 40) of the distribution area, depending upon the granularity,the displacement appearing between the position that may possibly becomethe distribution area reaching point AP#i represented by the PSF anglecomponent information before the rotation and the position that maypossibly become the distribution area reaching point AP#i represented bythe PSF angle component information after the rotation may have a badinfluence on the reproduction of light condensing characteristics of theemulation lens.

In particular, in the corresponding area (FIG. 44) represented by thePSF angle component information after the rotation, the PSF intensitydistribution sometimes overlaps partly with the PSF intensitydistribution of some other corresponding area or a portion of the PSFintensity distribution which should originally be included in a certaincorresponding area sometimes misses from the corresponding area.

Where the degree of the overlap or the missing of the PSF intensitydistribution is high, the reproducibility of light condensingcharacteristics of the emulation lens degrades.

Therefore, for the light condensing process, the ray rotation method canbe adopted.

However, if the granularity of the position, represented by thedistribution area reaching point AP#i, on the PSF intensity distributionreached by a ray is sufficiently fine, then even if the lens informationrotation method is used for the light condensing process, the lightcondensing characteristics of the emulation lens can be reproducedsufficiently accurately. Therefore, in this case, the lens informationrotation method can be adopted.

FIG. 64 is a block diagram depicting an example of a configuration ofthe light condensing processing section 38 that performs a lightcondensing process using lens information generated on the lensinformation generation axis.

It is to be noted that, in FIG. 64, like portions to those of FIG. 55are denoted by like reference numerals, and in the followingdescription, description of them is suitably omitted.

Referring to FIG. 64, the light condensing processing section 38includes a real space point selection section 141, an image formationvalue calculation section 242, a scale adjustment section 143, an imageformation position recognition section 244, an addition section 245 anda rotation processing section 246.

Accordingly, the light condensing processing section 38 of FIG. 64 iscommon to the case of FIG. 55 in that it includes the real space pointselection section 141 and the scale adjustment section 143.

However, the light condensing processing section 38 of FIG. 64 isdifferent from the case of FIG. 55 in that it includes the imageformation value calculation section 242, the image formation positionrecognition section 244 and the addition section 245 in place of theimage formation value calculation section 142, the image formationposition recognition section 144 and the addition section 145,respectively.

Further, the light condensing processing section 38 of FIG. 64 isdifferent from the case of FIG. 55 in that it newly includes therotation processing section 246.

The image formation value calculation section 242 determines adistribution area in which a distribution of image formation values ofrays emitted from a noticed real space point selected by the real spacepoint selection section 131 is recorded by the lens information rotationmethod or the ray rotation method, and supplies the distribution area tothe scale adjustment section 143.

In particular, the image formation value calculation section 242determines a distribution area in which a distribution of imageformation values of rays emitted from a noticed real space pointselected by the real space point selection section 131 from among rayssupplied from the rotation processing section 246 and rotated by acoincidence rotation angle −ALPHA is recorded using a PSF intensitydistribution and PSF angle component information supplied from therotation processing section 246 in accordance with the ray rotationmethod, and supplies the distribution area to the scale adjustmentsection 143.

Alternatively, the image formation value calculation section 242determines a distribution area in which a distribution of imageformation values of rays emitted from a noticed real space pointselected by the real space point selection section 131 and supplied fromthe rotation processing section 246 is recorded using a PSF intensitydistribution and PSF angle component information supplied from therotation processing section 246 and rotated by the coincidence rotationangle +ALPHA in accordance with the lens information rotation method,and supplies the distribution area to the scale adjustment section 143.

The image formation position recognition section 244 recognizes an imageplane shift position that is an image formation position on the virtualsensor at which rays passing through the emulation lens form an imagefrom image plane shift information supplied from the rotation processingsection 246 or the image plane shift information rotated by thecoincidence rotation angle +ALPHA, and supplies the image plane shiftposition to the addition section 245 together with the distribution areaafter adjustment of the scale from the scale adjustment section 143.

The addition section 245 performs a process similar to that of theaddition section 145 of FIG. 55.

In particular, the addition section 245 has a memory as the virtualsensor built therein and performs positioning of the distribution areaafter adjustment of the scale from the image formation positionrecognition section 244 and the virtual sensor depending upon an imageplane shift position from the image formation position recognitionsection 244.

Further, the addition section 245 adds the image formation valuesrecorded in the distribution area after the positioning with the virtualsensor on the virtual sensor in a unit of a pixel of the virtual sensor.

Then, the addition section 245 supplies an image, in which pixel valuesare given by an addition result of the image formation values obtainedon the virtual sensor, namely, on the memory, as an emulation image tothe display apparatus 13 (FIG. 1).

It is to be noted that, when a light condensing process by the lensinformation rotation method is performed, the addition section 245performs a process similar to that of the addition section 145 of FIG.55 as described above.

On the other hand, when a light condensing process by the ray rotationmethod is performed, the addition section 245 first rotates (the imageformation values recorded in) the distribution area after positioningreversely by the coincidence rotation angle −ALPHA when the ray has beenrotated by the rotation processing section 246, namely, by thecoincidence rotation angle +ALPHA, and then adds the image formationvalues recorded in the distribution area after the rotation on thevirtual sensor.

To the rotation processing section 246, rays are supplied from theincident ray reproduction section 36 (FIG. 3) and lens informationregarding the information points of the lens information generation axisis supplied from the emulation lens information generation section 34(FIG. 3).

When a light condensing process by the lens information rotation methodis performed, the rotation processing section 246 rotates the PSFintensity distribution, the PSF angle component information and theimage plane shift information from within lens information regarding theinformation points of the lens information generation axis from theemulation lens information generation section 34 by the coincidencerotation angle +ALPHA when the lens information generation axis isrotated such that a pixel of the virtual sensor corresponding to thenoticed real space point selected by the real space point selectionsection 141 comes to a pixel on the lens information generation axis.

Then, the rotation processing section 246 supplies the PSF intensitydistribution and the PSF angle component information after the rotationto the image formation value calculation section 242 and supplies theimage plane shift information after the rotation to the image formationposition recognition section 244.

Furthermore, the rotation processing section 246 supplies the rays fromthe incident ray reproduction section 36 as they are to the imageformation value calculation section 242 without rotating the same.

On the other hand, when a light condensing process by the ray rotationmethod is performed, the rotation processing section 246 rotates raysfrom the incident ray reproduction section 36 by the coincidencerotation angle −ALPHA when (a pixel of) the virtual sensor is rotatedsuch that a pixel of the virtual sensor corresponding to the noticedreal space point selected by the real space point selection section 141becomes a pixel on the lens information generation axis.

Then, the rotation processing section 246 supplies the rays after therotation to the image formation value calculation section 242.

Furthermore, the rotation processing section 246 supplies the PSFintensity distribution and the PSF angle component information fromwithin the lens information regarding the information points of the lensinformation generation axis from the emulation lens informationgeneration section 34 as they are to the image formation valuecalculation section 242 without rotating the same and supplies the imageplane shift information as it is to the image formation positionrecognition section 244 without rotating the same.

FIG. 65 is a flow chart illustrating an example of a light condensingprocess performed by the light condensing processing section 38 of FIG.64.

It is to be noted that, in the present embodiment, lens information isgenerated only for a noticed focus position in order to reduce theinformation amount of lens information as described hereinabove withreference to FIG. 61. Therefore, the light condensing process isperformed assuming that the focus position is set to the noticed focusposition when lens information is generated.

However, lens information can be generated in advance for each of theFmax focus positions (FIG. 37). In this case, the light condensingprocessing section 38 sets a noticed focus position and can perform alight condensing process using lens information regarding the noticedfocus position.

Alternatively, the emulation lens information generation section 37 cangenerate lens information for a noticed focus position after the noticedfocus position is set by the light condensing processing section 38.

Here, in FIG. 65, it is assumed that a light condensing process by theray rotation method is performed.

At steps S251 to S254, processes similar to those at steps S151 to S154of FIG. 56 are performed.

In particular, the real space point selection section 141 acquires amultilayer parallax map supplied from the parallax informationgeneration section 31 at step S251 and selects one pixel that has notbeen selected as a noticed pixel as yet from among the pixels of thereference image HD1 as a noticed pixel at step S252.

Further, at step S253, the real space point selection section 141selects one parallax that has not been selected as a noticed parallax asyet from among the parallaxes of the noticed pixel registered in themultilayer parallax map from the parallax information generation section31 as a noticed parallax. Then at step S254, the real space pointselection section 141 selects a real space point corresponding to thenoticed pixel having the noticed parallax as a noticed real space point.

Then, the processing advances from step S254 to step S255, at which therotation processing section 246 calculates the coincidence rotationangle −ALPHA when the virtual sensor is rotated such that a pixel of thevirtual sensor corresponding to the noticed real space point selected bythe real space point selection section 141 becomes a pixel on the lensinformation generation axis. Then, the processing advances to step S256.

At step S256, the rotation processing section 246 selects one lens areaunit that has not been selected as a noticed lens area unit as yet fromamong the lens area units of the emulation lens as a noticed lens areaunit. Then, the processing advances to step S257.

At step S257, the rotation processing section 246 acquires a ray headingfrom the noticed real space point toward the noticed lens area unit fromamong rays supplied from the incident ray reproduction section 36 as anoticed ray. Then, the processing advances to step S258.

At step S258, the rotation processing section 246 rotates the noticedray by the coincidence rotation angle −ALPHA calculated at step S255around the optical axis and supplies the noticed ray after the rotationto the image formation value calculation section 242. Then, theprocessing advances to step S259.

At step S259, the image formation value calculation section 242 decideswhether or not the noticed ray after the rotation from the rotationprocessing section 246 reaches the emulation lens from the noticed realspace point.

If it is decided at step S259 that the noticed ray after the rotationreaches the emulation lens, namely, if the parallax allocated to thenoticed ray after the rotation (parallax allocated by the incident rayreproduction process described hereinabove with reference to FIGS. 33 to35) is equal to the noticed parallax, then the processing advances tostep S260.

At step S260, for the noticed ray after rotation decided to reach theemulation lens by the image formation value calculation section 242, thescale adjustment section 143, the image formation position recognitionsection 244 and the addition section 245, namely, for the noticed rayafter rotation obtained by rotating the noticed ray that remains as aresult of the collision decision, a ray addition process hereinafterdescribed is performed. Then, the processing advances to step S261.

On the other hand, if it is decided at step S259 that the noticed rayafter rotation does not reach the emulation lens, namely, if theparallax allocated to the noticed ray after rotation is not equal to thenoticed parallax, then the processing skips step S260 and advances tostep S261. Accordingly, when the noticed ray after rotation does notreach the emulation lens, the ray addition process is not performed forthe noticed ray after rotation.

At steps S261 to S263, processes similar to those at steps S159 to S161of FIG. 56 are performed.

In particular, at step S261, the rotation processing section 246 decideswhether or not all of the lens area units of the emulation lens havebeen selected as a noticed lens area unit. Then, if it is decided thatall of the lens area units of the emulation lens have not been selectedas a noticed lens area unit, then the processing returns to step S256,and thereafter, similar processes are repeated.

On the other hand, if it is decided at step S261 that all of the lensarea units of the emulation lens have been selected as a noticed lensarea unit, then the processing advances to step S262, at which the realspace point selection section 141 decides whether or not all of theparallaxes of the noticed pixel registered in the multilayer parallaxmap have been selected as a noticed parallax.

If it is decided at step S262 that all of the parallaxes of the noticedpixel registered in the multilayer parallax map have not been selectedas a noticed parallax as yet, then the processing returns to step S253,and thereafter, similar processes are repeated.

On the other hand, if it is decided at step S262 that all of theparallaxes of the noticed pixel registered in the multilayer parallaxmap have been selected as a noticed parallax, then the processingadvances to step S263, at which the real space point selection section141 decides whether or not all of the pixels of the reference image HD1have been selected as a noticed pixel.

If it is decided at step S263 that all of the pixels of the referenceimage HD1 have not been selected as a noticed pixel, then the processingreturns to step S252, and thereafter, similar processes are repeated.

On the other hand, if it is decided at step S263 that all of the pixelsof the reference image HD1 have been selected as a noticed pixel, thenthe addition section 245 supplies an image, in which the pixel valuesare provided by addition results of the image formation values on thevirtual sensor, obtained by the processes till then as an emulationimage to the display apparatus 13 (FIG. 1), thereby ending the lightcondensing process.

FIG. 66 is a flow chart illustrating an example of the ray additionprocess performed at step S260 of FIG. 65.

At step S271, the image formation value calculation section 242 detectsa corresponding information point, which is an information point nearestto a pixel (position) after rotation of the virtual sensor correspondingto a noticed real space point when the pixel is rotated by thecoincidence rotation angle −ALPHA calculated at step S255 of FIG. 65,from among the information points of the lens information generationaxis.

Further, the image formation value calculation section 242 acquires lensinformation regarding a real space point corresponding to thecorresponding information point and the noticed parallax from within thelens information regarding the information points of the lensinformation generation axis from the rotation processing section 246.

Then, the image formation value calculation section 242 determines acorresponding area that is a position on the distribution area in whicha PSF intensity distribution in which a noticed ray after rotation fromthe rotation processing section 246 reaches is recorded using (adistribution area having recorded therein) a PSF intensity distributionand the PSF angle component information regarding the noticed focusposition f from within the lens information regarding a real space pointcorresponding to the corresponding information point and the noticedparallax.

Further, the image formation value calculation section 242 determinesthe product of the PSF intensity distribution of the corresponding areaand the luminance allocated to the noticed ray after rotation (luminanceallocated by the incident ray reproduction process described hereinabovewith reference to FIGS. 33 to 35) as (a distribution of) an imageformation value of the noticed ray after rotation.

Then, the image formation value calculation section 242 supplies thedistribution area in which the image formation value of the noticed rayafter rotation is recorded to the scale adjustment section 143.Thereafter, the processing advances from step S271 to step S272.

At step S272, the scale adjustment section 143 acquires lens informationregarding the real space point corresponding to the correspondinginformation point and the noticed parallax from within the lensinformation regarding the information points of the lens informationgeneration axis from the emulation lens information generation section34.

Then, the scale adjustment section 143 reduces or expands thedistribution area in which the distribution of image formation valuesfrom the image formation value calculation section 242 is recorded usingthe image plane pitch regarding the noticed focus position f from withinthe lens information regarding the real space point corresponding to thecorresponding information point and the noticed parallax to adjust thescale of the distribution area so as to coincide with the scale of thevirtual sensor.

Further, the scale adjustment section 143 supplies the distribution areaafter the adjustment of the scale to the addition section 245 throughthe image formation position recognition section 244. Thereafter, theprocessing advances from step S272 to step S273.

At step S273, the image formation position recognition section 244acquires lens information regarding the real space point correspondingto the corresponding information point and the noticed parallax fromwithin the lens information regarding the information point of the lensinformation generation axis from the rotation processing section 246.

Then, the image formation position recognition section 244 recognizesthe image plane shift position that is an image formation position onthe virtual sensor at which a noticed ray before rotation forms an imagethrough the emulation lens from the image plane shift informationregarding the noticed focus position f and the coincidence rotationangle −ALPHA by which the noticed ray is rotated by the rotationprocessing section 246 from within the lens information regarding a realspace point corresponding to the corresponding information point and thenoticed parallax and supplies the image plane shift position to theaddition section 245. Then, the processing advances to step S274.

At step S274, the addition section 245 performs positioning of thedistribution area after adjustment of the scale obtained by the scaleadjustment section 143 and the virtual sensor depending upon the imageplane shift position from the image formation position recognitionsection 244.

In particular, the addition section 245 performs positioning of thedistribution area after adjustment of the scale and the virtual sensorsuch that the center point CP (FIG. 54) of the distribution area afteradjustment of the scale and the image plane shift position of thevirtual sensor coincide with each other.

Further, at step S274, the addition section 245 rotates (the imageformation values recorded in) the distribution area after positioningreversely around the optical axis (or the center point CP) by thecoincidence rotation angle −ALPHA when the noticed ray is rotated by therotation processing section 246, namely, around the optical axis (or thecenter point CP) by the coincidence rotation angle +ALPHA. Then, theprocessing advances to step S275.

At step S275, the addition section 245 adds the image formation valuesrecorded in the distribution area after rotation on the virtual sensorin a unit of a pixel of the virtual sensor. That is, the additionsection 245 adds storage values of the memory as the virtual sensor andthe image formation values and rewrites the storage values of the memorywith the addition values obtained as a result of the addition.

The ray addition process ends therewith, and the processing returns.

As described above, where lens information is generated for real spacepoints corresponding to a plurality of information points that are aplurality of positions of part of the plane of the virtual sensor and alight condensing process is performed using the lens information, a blurdegree or other light condensing characteristics of the emulation lenscan be reproduced accurately by a reduced data amount.

It is to be noted that, while, in the present embodiment, a line segment(FIG. 59) that has a length equal to 1/2 the length of the diagonal ofthe virtual sensor and extends in the vertical direction from the centerof the virtual sensor is adopted as the lens information generationaxis, as the lens information generation axis, an arbitrary one axisextending from the center of the virtual sensor can be adopted.

In particular, as the lens information generation axis, for example, aline segment interconnecting the center of the virtual sensor and onepixel of a diagonal of the virtual sensor can be adopted.

Further, the plurality of information points are not limited to aplurality of points that equally divide the lens information generationaxis extending from the center of the virtual sensor.

In particular, as the plurality of information points, for example, aplurality of points arranged on a straight line or not arranged on astraight line where the image height of the plane of the virtual sensordiffers can be adopted. It is to be noted that image heights of aplurality of information points are preferably scattered evenly within arange from zero to a maximum value of the image height of the virtualsensor.

FIG. 67 is a view illustrating a method for determining, where a lightcondensing process by the ray rotation method (FIGS. 65 and 66) isperformed, an image plane shift position that is an image formationposition on the virtual sensor at which rays before rotation form animage (a method for recognizing the image formation position at stepS273 in FIG. 66).

Here, it is assumed that the image plane shift position is represented,for example, by a coordinate (x, y) of a two-dimensional coordinatesystem in which the origin is a pixel at the left upper corner of thevirtual sensor and the pixel pitch of the virtual sensor is 1.

Further, it is assumed that the image plane shift information representsdistances (micrometer) in the x direction and the y direction from theoptical axis (center of the virtual sensor) SO of the image plane shiftposition.

The image formation position recognition section 244 recognizes an imageplane shift position that is an image formation position on the virtualsensor at which a noticed ray before rotation forms an image (theposition is hereinafter referred to also as image plane shift positionbefore rotation) from the image plane shift information regarding thereal space point corresponding to the corresponding information pointhaving the noticed parallax and the coincidence rotation angle −ALPHA bywhich the noticed ray has been rotated by the rotation processingsection 246 as described hereinabove in connection with step S273 ofFIG. 66.

Here, the corresponding information point is a point on the sensorinformation generation axis extending in an upward direction from thecenter SO of the virtual sensor, and image plane shift informationimg_height regarding a real space point corresponding to such acorresponding information point and the noticed parallax represents adistance from the center SO of the virtual sensor to a position A on astraight line that passes the center SO and extends in a verticaldirection. The position A is a position on the virtual sensor reached bya principal ray, which is emitted from the real space pointcorresponding to the corresponding information point and the noticedparallax, through the emulation lens.

Now, it is assumed that the horizontal width of the virtual sensor(distance between a pixel at the left end and a pixel at the right end)is denoted as width and the pixel pitch of the virtual sensor (distancebetween adjacent pixels) is denoted as Sensor pitch.

The image plane shift position before rotation that is an imageformation position on the virtual sensor at which a noticed ray beforerotation forms an image is a position when the position A is rotatedreversely by the coincidence rotation angle −ALPHA when the noticed rayhas been rotated, namely, a position rotated by the coincidence rotationangle +ALPHA.

Accordingly, the x coordinate X of the image plane shift position beforerotation can be determined in accordance with an expressionX=width/2+(img_height×cos(90 degrees−ALPHA))/Sensor pitch.

Also the y coordinate of the image plane shift position before rotationcan be determined similarly.

The image formation position recognition section 244 determines andrecognizes (the x coordinate and the y coordinate of) the image planeshift position before rotation in such a manner as described above.

Then, the addition section 245 performs positioning of the distributionarea after adjustment of the scale and the virtual sensor depending uponthe image plane shift position before rotation obtained by reverselyrotating the position (image plane shift position) A represented by theimage plane shift information img_height regarding the real space pointcorresponding to the corresponding information point and the noticedparallax by the coincidence rotation angle −ALPHA when the noticed rayis rotated (by rotating the position A by the coincidence rotation angle+ALPHA).

In particular, the addition section 245 performs positioning of thedistribution area after adjustment of the scale and the virtual sensorsuch that the center point CP (FIG. 54) of the distribution area afteradjustment of the scale and the image plane shift position beforerotation of the virtual sensor coincide with each other.

It is to be noted that, as described above, since the image plane shiftposition before rotation is determined by rotating the position (imageplane shift position) represented by the image plane shift informationimg_height regarding the real space point corresponding to thecorresponding information point and the noticed parallax, a roundingerror arising from the rotation occurs.

By this rounding error, the accuracy of the image plane shift positionbefore rotation is degraded, and arising from the degradation of theaccuracy of the image plane shift position before rotation, thereproducibility of the light condensing characteristics of the emulationlens in the light condensing process sometimes degrades.

Therefore, in order to suppress the degradation of the accuracy of theimage plane shift position before rotation, interpolation can beperformed for the (image plane shift position represented by) the imageplane shift information.

As interpolation of (an image plane shift position represented by) imageplane shift information, interpolation in a direction of arrangement ofinformation points (direction perpendicular to the optical axis)(direction of the image height) and interpolation in the parallaxdirection are available.

FIG. 68 is a view illustrating interpolation in a direction ofarrangement of the information points of (the image plane shift positionrepresented by) the image plane shift information.

The light condensing process by the ray rotation method is performedusing lens information regarding a real space point corresponding to acorresponding information point and a noticed parallax, which is aninformation point nearest to a pixel (position) after rotation when apixel (position) of the virtual sensor corresponding to a noticed realspace point is rotated by the coincidence rotation angle −ALPHA (suchlens information is hereinafter referred to merely as lens informationregarding a corresponding information point).

In FIG. 68, a pixel (position) pp′ after rotation when a pixel(position) pp of the virtual sensor corresponding to a noticed realspace point is rotated by the coincidence rotation angle −ALPHA ispositioned between adjacent information points A and B, and acorresponding information point that is an information point nearest tothe pixel pp′ after the rotation is an information point B.

In this case, as the interpolation in the direction of arrangement ofthe information points, interpolation in which image plane shiftpositions ih1 and ih2 regarding the information points A and B betweenwhich the pixel pp′ after the rotation is sandwiched are used isperformed, for example, in accordance with the ratio between thedistances a and b from the pixel pp′ after the rotation to theinformation points A and B, respectively.

In particular, in the interpolation in the direction of arrangement ofthe information points, the image plane shift position ih as aninterpolation value in the direction of arrangement of the informationpoints is determined, for example, in accordance with an expressionih=(ih1×b+ih2×a)/(a+b).

Then, the image plane shift position ih is used as the interpolationvalue in the direction of arrangement of the information points in placeof the image plane shift position ih2 regarding the information point Bthat is the corresponding information point to perform positioning ofimage formation positions of rays emitted from the noticed real spacepoint.

FIG. 69 is a view illustrating interpolation in the parallax directionof (the image plane shift position represented by) image plane shiftinformation.

In the present embodiment, a parallax is determined, for example, withthe accuracy of 1/4 pixel as described hereinabove with reference toFIG. 6, and the parallax of the accuracy of 1/4 pixel is integrated andused.

Therefore, although, for example, in FIG. 32 and so forth, the parallaxd that can be registered into a parallax map is DPN=Dmax−Dmin+1 integralvalues of 1 pixel increments from the minimum value Dmin to the maximumvalue Dmax, it is possible to register parallaxes of the accuracy of 1/4pixel in a parallax map and, when a parallax registered in the parallaxmap is to be used, integrate the parallax.

As described above, where parallaxes of an accuracy equal to or lowerthan the accuracy of a pixel such as the accuracy of 1/4 pixel areregistered in a parallax map, in interpolation of image plane shiftinformation in the parallax direction, a parallax of an accuracy equalto or lower than the accuracy of a pixel registered in the parallax mapcan be used as it is without integrating the same.

For example, it is assumed now that parallaxes of the accuracy of 1/4pixel are registered in a parallax map.

In the emulation lens information generation process of FIG. 61, (animage plane shift position represented by) image plane shift informationas lens information regarding a noticed real space point correspondingto a noticed information point and a noticed parallax is determined byselecting a parallax that can be registered into the parallax map as anoticed parallax.

As parallaxes that can be registered into a parallax map and areselected as a noticed parallax by the emulation lens informationgeneration process, DPN=Dmax−Dmin+1 integral values of 1 pixelincrements from the minimum value Dmin to the maximum value Dmax areused.

Accordingly, in the emulation lens information generation process, animage plane shift position is determined for a parallax of an integralvalue.

In FIG. 69, a parallax of a next magnitude smaller than a parallax D ofa certain integral value has an integral value D+1. Then, an image planeshift position ih1 is determined for the parallax D of an integralvalue, and another image plane shift position ih2 is determined for theparallax D+1 of an integral value of the next magnitude.

On the other hand, in the light condensing process of FIG. 65, as theparallax of a noticed pixel selected from within a reference image, anoticed parallax is selected from among parallaxes registered in aparallax map (step S253).

In this case, the parallaxes registered in the parallax map areintegrated and selected as a noticed parallax.

Then, in the light condensing process, the ray addition process (FIG.66) is performed using lens information regarding a real space pointcorresponding to a corresponding information point and the noticedparallax.

For (the image plane shift position represented by) the image planeshift information from within the lens information used in the rayaddition process, interpolation in the parallax direction can beperformed using the parallaxes registered in the parallax map withoutintegrating them.

In particular, in FIG. 69, the parallax (noticed parallax) of the realspace point corresponding to the corresponding information point and thenoticed parallax is D+0.75 that is a parallax of the accuracy of 1/4pixel.

Here, in the case where the parallax registered in the parallax map is aparallax of the accuracy of 1/4 pixel, as a parallax of the accuracy of1/4 pixel from the parallax D of an integral value to the parallax D+1of a next integral value, D+0.25, D+0.5 and D+0.75 are available.

When a parallax registered in the parallax map is integrated and used,if it is assumed that the integration is performed, for example, bytruncation after the decimal point, then where the noticed parallax isD, D+0.25, D+0.5 or D+0.75, they are all integrated into the integralvalue D.

Then, the ray addition process (FIG. 66) is performed using the imageplane shift position ih1 for the noticed parallax integrated to theintegral value D.

As interpolation in the parallax direction, interpolation in which theimage plane shift positions ih1 and ih2 for the parallaxes D and D+1 ofintegral values between which the noticed parallax D+0.75 that has notbeen integrated is sandwiched are used is performed in accordance with aratio between the distances of 0.75 and 0.25 from the noticed parallaxD+0.75 to the parallaxes D and D+1 of integral values.

In particular, in the interpolation in the parallax direction, an imageplane shift position ih as an interpolation value in the parallaxdirection is determined in accordance with an expressionih=ih1×0.25+ih2×0.75.

Then, in place of the image plane shift position ih1 for the noticedparallax integrated to the integer value D, the image plane shiftposition ih as an interpolation value in the parallax direction is usedto perform (positioning of an image formation position of a ray fromwithin) the ray addition process.

Where such interpolation of (the image plane shift position representedby) image plane shift information is performed, the reproducibility ofthe light condensing characteristics of the emulation lens in the lightcondensing process can be improved.

It is to be noted that, for image plane shift information, it ispossible to perform only one of the interpolation in the arrangementdirection of information points and the interpolation in the parallaxdirection or perform both of them.

<Emulation Result>

FIG. 70 is a view depicting an example of an emulation image obtained asa result of a lens emulation process by the lens emulation section 35(FIG. 3).

FIG. 70 depicts an emulation image obtained from a picked up image whenan image of a real space in which a bar-like object obj2 is disposed atthe front side of an object obj1 is taken.

According to the emulation image, it can be confirmed that rays emittedfrom a portion of the object obj1 hidden by the bar-like object obj2collide with the object obj2 and cannot be viewed.

FIG. 71 is a view depicting another example of the emulation imageobtained as a result of a lens simulation process by the lens emulationsection 35.

It is to be noted that the emulation image of FIG. 71 is an emulationimage obtained from a picked up image obtained by picking up an image ofa real space in which a bar-like object obj2 is disposed at the frontside of an object obj1 similarly to FIG. 70.

A of FIG. 71 depicts an emulation image where, as the image plane shiftposition, not a position on the virtual sensor at which a principal rayreaches through the emulation lens but a position on the virtual sensorat which a principal ray reaches through the virtual lens is adopted.

Where, in the light condensing process, as the image plane shiftposition, not a position on the virtual sensor at which a principal rayreaches through the emulation lens but a position on the virtual sensorat which a principal ray reaches through the virtual lens is adopted, anerror arising from displacement in position of the input pupil betweenthe virtual lens and the emulation lens occurs with the position of thevirtual sensor to which an image formation value of a principal ray isto be added. Therefore, in A of FIG. 71, in the emulation image, aportion of the object obj1 at the interior side, which should be hiddenby the bar-like object obj2 and not be able to be viewed, is viewed.

B of FIG. 71 depicts an emulation image where interpolation of imageplane shift information is not performed.

In B of FIG. 71, it can be confirmed that, by an influence of a roundingerror arising from rotation (of the image plane shift positionrepresented by) the image plane shift information, a line segment thatforms a profile of the bar-like object obj2 and extends in the verticaldirection is uneven.

By adopting a position on the virtual sensor at which a principal rayreaches through the emulation lens as the image plane shift position orby performing interpolation of image plane shift information, such astate as depicted in FIG. 71 can be prevented from occurring in anemulation image.

<Explanation of Computer to which Present Technology is Applied>

While the series of processes described above can be executed byhardware, it may otherwise be executed by software. In the case wherethe series of processes is executed by software, a program whichconstructs the software is installed into a computer for universal useor the like.

FIG. 72 is a block diagram depicting an example of a configuration ofone embodiment of a computer into which the program for executing theseries of processes described above is installed.

The program can be recorded in advance into a hard disk 405 or a ROM(Read Only Memory) 403 as a recording medium built in the computer.

Alternatively, the program can be stored (recorded) in a removablerecording medium 411. Such a removable recording medium 411 as justdescribed can be provided as so-called package software. Here, as theremovable recording medium 411, for example, a flexible disk, a CD-ROM(Compact Disc Read Only Memory), an MO (Magneto Optical) disk, a DVD(Digital Versatile Disc), a magnetic disk, a semiconductor memory and soforth are available.

It is to be noted that, in addition to installation from such aremovable recording medium 411 as described above into the computer, theprogram can be downloaded into the computer through a communicationnetwork or a broadcasting network and installed into the hard disk 405built in the computer. In particular, the program can be transferred,for example, from a download site by wireless transfer to the computerthrough an artificial satellite for digital satellite broadcasting or bywired transfer to the computer through a network such as a LAN (LocalArea Network) or the Internet.

The computer has a CPU (Central Processing Unit) 402 built therein, andan input/output interface 410 is connected to the CPU 402 through a bus401.

If an inputting section 407 is operated by a user or the like to inputan instruction to the CPU 402 through the input/output interface 410,then a program stored in the ROM (Read Only Memory) 403 is executed.Alternatively, the CPU 402 loads the program stored in the hard disk 405into a RAM (Random Access Memory) 404 and executes the program.

Consequently, the CPU 402 performs a process in accordance with the flowcharts described hereinabove or performs a process carried out by theconfiguration of the block diagram described above. Then, the CPU 402outputs a result of the process, for example, from an outputting section406 through the input/output interface 410, transmits the result of theprocess from a communication section 408, or causes the result of theprocess to be recorded on the hard disk 405 or the like as occasiondemands.

It is to be noted that the inputting section 407 is configured from akeyboard, a mouse, a microphone and so forth. Meanwhile, the outputtingsection 406 is configured from an LCD (Liquid Crystal Display), aspeaker and so forth.

Here, in the present specification, the processes performed inaccordance with the program by the computer need not be carried out in atime series in the order as described in the flow charts. In particular,the processes to be executed in accordance with the program by thecomputer include processes executed in parallel or individually (forexample, processes by parallel processing or by an object).

Further, the program may be processed by a single computer (processor)or may be processed in a distributed manner by a plurality of computers.Further, the program may be transferred to and executed by a remotecomputer.

Further, in the present specification, the term system signifies anaggregation of a plurality of components (devices, modules (parts) orthe like) and all components may or may not be accommodated in the samehousing. Accordingly, a plurality of apparatus accommodated in separatehousings and connected to each other through a network and also oneapparatus wherein a plurality of modules are accommodated in a singlehousing are systems.

It is to be noted that the embodiment of the present technology is notrestricted to the embodiment described above and can be altered invarious manners without departing from the subject matter of the presenttechnology.

For example, the present technology can assume a configuration for cloudcomputing wherein one function is shared and processed cooperatively bya plurality of apparatus through a network.

Further, the steps described hereinabove in connection with the flowcharts can be executed by a single apparatus or can be executed in ashared manner by a plurality of apparatus.

Furthermore, where a plurality of processes are included in one step,the plurality of processes included in the one step may be executed byone apparatus or may be executed in a shared manner by a plurality ofapparatus.

Further, the effects described in the present specification areillustrative to the end and are not restrictive, and other effects maybe exhibited.

It is to be noted that the present technology can assume suchconfigurations as described below.

<1>

An image processing apparatus, including:

a ray generation section configured to generate rays to be incident to avirtual lens having a synthetic aperture configured from a plurality ofimage pickup sections that pick up images of a plurality of visualpoints from a real space point in a real space; and

a luminance allocation section configured to allocate a luminance torays remaining as a result of a collision decision for deciding whetheror not the rays generated by the ray generation section collide with anobject before the rays are incident to the virtual lens.

<2>

The image processing apparatus according to <1>, in which the raygeneration section

divides the virtual lens into lens area units that are small regions,and

generates rays to be incident to the lens area units from the realspace.

<3>

The image processing apparatus according to <1> or <2>, furtherincluding:

a collision decision section configured to perform the collisiondecision.

<4>

The image processing apparatus according to <3>, in which the collisiondecision section performs the collision decision using a parallax map inwhich a parallax of a reference image that is one of the images of theplurality of visual points from the other image or images is registered.

<5>

The image processing apparatus according to <4>, in which the collisiondecision section

determines a cross point between each of the rays and a parallax planethat is positioned at a distance corresponding to a decision parallaxfor the collision decision and is orthogonal to an optical axis of thevirtual lens, and performs the collision decision depending upon whetheror not a parallax of a cross point pixel of the reference imagecorresponding to the cross point coincides with the decision parallax.

<6>

The image processing apparatus according to <5>, in which the collisiondecision section

sets a value of a parallax within a range of a maximum value of theparallax to the parallax corresponding to the depth of the real spacepoint as the decision parallax, and decides, where the parallax of thecross point pixel does not coincide with the decision parallax, that theray does not collide.

<7>

The image processing apparatus according to <6>, in which the collisiondecision section

successively sets a value decreasing from the maximum value of theparallax as the decision parallax, and

decides, where the parallax of the cross point pixel does not coincidewith the decision parallax until the distance corresponding to thedecision parallax comes to coincide with the depth of the real spacepoint, that the ray does not collide.

<8>

The image processing apparatus according to any one of <1> to <7>, inwhich

the luminance allocation section allocates a luminance to the rays usinga pixel value of a pixel of a parallax equal to the parallaxcorresponding to the depth of the real space point from among the pixelsof the images of the plurality of visual points corresponding to thereal space point.

<9>

An image processing method, including:

generating rays to be incident to a virtual lens having a syntheticaperture configured from a plurality of image pickup sections that pickup images of a plurality of visual points from a real space point in areal space; and

allocating a luminance to rays remaining as a result of a collisiondecision for deciding whether or not the rays collide with an objectbefore the rays are incident to the virtual lens.

REFERENCE SIGNS LIST

-   11 Image pickup apparatus, 12 Image processing apparatus, 13 Display    apparatus, 21 ₁ to 21 ₇ Camera unit, 31 Parallax information    generation section, 32 Calibration data acquisition section, 33    Saturated pixel restoration section, 34 Lens design data acquisition    section, 35 Lens emulation section, 36 Incident ray reproduction    section, 37 Emulation lens information generation section, 38 Light    condensing processing section, 41 Reference parallax map generation    section, 42 Multilayer parallax map generation section, 51    Saturation decision section, 52 Restoration section, 61 Standard    luminance picked up image generation section, 62 Low luminance    picked up image generation section, 71 Parallax information    acquisition section, 72 Standard luminance picked up image    generation section, 73 Low luminance picked up image generation    section, 74 Saturation decision section, 75 Restoration section, 101    Real space point selection section, 102 Ray generation section, 103    Collision decision section, 104 Luminance allocation section, 131    Real space point selection section, 132 Information calculation    section, 133 Focus position selection section, 141 Real space point    selection section, 142 Image formation value calculation section,    143 Scale adjustment section, 144 Image formation position    recognition section, 145 Addition section, 231 Real space point    selection section, 242 Image formation value calculation section,    244 Image formation position recognition section, 245 Addition    section, 246 Rotation processing section, 401 Bus, 402 CPU, 403 ROM,    404 RAM, 405 Hard disk, 406 Outputting section, 407 Inputting    section, 408 Communication section, 409 Drive, 410 input/output    interface, 411 Removable recording medium

1. An image processing apparatus, comprising: a ray generation sectionconfigured to generate rays to be incident to a virtual lens having asynthetic aperture configured from a plurality of image pickup sectionsthat pick up images of a plurality of visual points from a real spacepoint in a real space; and a luminance allocation section configured toallocate a luminance to rays remaining as a result of a collisiondecision for deciding whether or not the rays generated by the raygeneration section collide with an object before the rays are incidentto the virtual lens.
 2. The image processing apparatus according toclaim 1, wherein the ray generation section divides the virtual lensinto lens area units that are small regions, and generates rays to beincident to the lens area units from the real space.
 3. The imageprocessing apparatus according to claim 1, further comprising: acollision decision section configured to perform the collision decision.4. The image processing apparatus according to claim 3, wherein thecollision decision section performs the collision decision using aparallax map in which a parallax of a reference image that is one of theimages of the plurality of visual points from the other image or imagesis registered.
 5. The image processing apparatus according to claim 4,wherein the collision decision section determines a cross point betweeneach of the rays and a parallax plane that is positioned at a distancecorresponding to a decision parallax for the collision decision and isorthogonal to an optical axis of the virtual lens, and performs thecollision decision depending upon whether or not a parallax of a crosspoint pixel of the reference image corresponding to the cross pointcoincides with the decision parallax.
 6. The image processing apparatusaccording to claim 5, wherein the collision decision section sets avalue of a parallax within a range of a maximum value of the parallax tothe parallax corresponding to the depth of the real space point as thedecision parallax, and decides, where the parallax of the cross pointpixel does not coincide with the decision parallax, that the ray doesnot collide.
 7. The image processing apparatus according to claim 6,wherein the collision decision section successively sets a valuedecreasing from the maximum value of the parallax as the decisionparallax, and decides, where the parallax of the cross point pixel doesnot coincide with the decision parallax until the distance correspondingto the decision parallax comes to coincide with the depth of the realspace point, that the ray does not collide.
 8. The image processingapparatus according to claim 1, wherein the luminance allocation sectionallocates a luminance to the rays using a pixel value of a pixel of aparallax equal to the parallax corresponding to the depth of the realspace point from among the pixels of the images of the plurality ofvisual points corresponding to the real space point.
 9. An imageprocessing method, comprising: generating rays to be incident to avirtual lens having a synthetic aperture configured from a plurality ofimage pickup sections that pick up images of a plurality of visualpoints from a real space point in a real space; and allocating aluminance to rays remaining as a result of a collision decision fordeciding whether or not the rays collide with an object before the raysare incident to the virtual lens.